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ST.\NDARD$ ^T) PROCEDURES DEVELOPMENT AND -IMPLEME^TAT-ION 

OF AN EVALUATION CONTRACT 



* * Introduction , '* , • 

As one loojcs at eddcation at the nationsfl level, it is .dif^i^lt to 
avoid being pessimistic about the future of genuine edjiCatior^l improve- 
^ent^iji the nation's schools. The field of education lacks an abiding 
tradition *of carefully planned changes or systematic injquiry into the, 
effectiveness of new programs, processes and products which are inserted ' 
into ther.classVoom. As a result, many of the changes 'occurring in educa- 
tigv, are lit^tle more than random adoption of faddish innovations: Probably 
the greatest contributors to such a situation are the lack of dependable 
inro^matisin about the performance of educational prpducts, practices and 
orograms and the absence pf esta'blished systems and procedures f<?r pVo- 
ducing 'suoh. informatio'h. ^ ' . , ' 

Altliough not sufficient in and of itself, ^dudational evaluation.^ 
would seem to hold greater promisee than any other approach ^ to providing 
educators with information they need to h^lp improve the process of - 
education. Recognition of this fact has led many educational and legis- 
lative leaders throughout the nation to turn to evaluation (arid related 
activities such ^s assessment and accountability) as the key to improving 
the quality of education in their state or district?^ 

In Alaska, the importance of evaluation has been recognized at 
, several levels. In addition to the evaluation 'stiidies mandated^-under 
guidelines far federally funded, projects- in several categorical areas, 
evaluation activities thave' been initiated by the Ala^a'^-'Depattment of 
Education and by iQcal education agencies* in several ar\as .of the state. 



Hdwever, ch*e success of ; these efforts has b6en somewhat j^limit;ed to date 



and the general » quality ^f evaluations conducted throughout 'the State of 



Alaska, especially those in the rural areas, has been rather unsatisf actoty 
Obviously there are notable exceptions to this statement, but it does seem 
to be true of a majority .of the federally sponsor.ed evaluatfions as well as 
a majority of the evaluative 'studies which hav^ been sponsored ,by . \ 
..individual schools ©r school districts. 

Many reasons for the generally low.'quality Of evaluation studies 
might be postulated. Several general Impediments have 'been discussed in . 
.detail' elsewhere (Worthen, 1972a) /and 'i^tll not be reiterated h.ere. Three 

inadequacies which contribute to-^the poor quality of most^'.educational 

• / ^ ^ ,1 • . 

evaluation studies should be singled out for some elabpratioh' here, iit>wever 
since they have- been, specif ically listed by a st-aff member of the- Alaska- 
Uepartnent .of Educatian as factors which he proposes as among the chief 
causes of the '(insatisf actory quality of most evaluation studied conducted 
.in Alas lea. j ^ - , , 

The ^three hypothesized inadequacies are: 

1. The l^ck' of .explicit standards and procedures f of *use in 
y> * ' developing atid implementing an evaluation contract; 

e - ' . * 

2» Th4 lack of olearcut criteria for determining the adequacy 
of .an evaluation design or study; and 

3', the lack of 'standards fot ethical conduct an4 practice in , 
evaluation which could^ serve to guide persons and agencies 
* • . * who conduct Valuation '^^tudies in education. 



■^Alaska is by no-means uniqui in this regard. ^ This statement would 

'(^escribe .almost any other* state in the union at least as well. 

<i • » 

''^The points are abstracted from a 1974 discussion , document prepared^ 
by Frank N'e-lson, entitled ^'Criteria for Educational Evaluations./*. 




This paper is an'explidit a^empt to, deal with 'the ^irst ina'dequ'^cy , 
lac'k of -^staniiirds and prpcedur^s 'f or evaluation contracts. ^The sedond 
topic, meta-ev|aluation cri^fia, is the subject of ^a^^e'parate ,paper'^' an4 , * 
will not bel^dealc wllthi^ere.l ^he third topic, the prpfessional ethics 



of the educa-tional 



^valuator,! is' a perv4siv,e ^element which/*undergirds and' 



overar-ches^.both of the previous - topic sX^The -need' for a code of ,ethics ^ 
for professional evaluators ^has been .argue^elsewhere (Wright, 1975), &u»t 
specification of such a code *is ^a task of *suci\ magnitude that it far * 
transcends the scope off the present; paper on contract specifications or 
of its , companion pamper on criteria for. evaluation designs and studies. 
Therefore, although the issue of 'ethical conduct is' obviously implicit 
in much of what is presented in both papers, it should be i^nderstood that ^ 
neither represents an attempt to discuss or develop explic-it standards* 
for etliical conduct or'practice of educational evaluation. That' effort 
badly needed in tKe opinions of the 'present authors, mu^t await a more 
comorehensive discussion which can focus spjecif ically ^on that topic. 

' • I '- .'X ■ ■ " ■ . ' - , • " 

Therefore, this- pape'r addresses only the first hypothesized inadequacy — • 
the need for exblicit standards and procedures for evaluation contracting. 



( 



^The Ala; 



ka Department of Education has contracted .with the* Nor thwest 
egional. Educational Laboratory for a paper on' x;riteria* f or use in^ evalua- 
ting Evaluation designs and Studies. THat 'paper, authored by James R. ' , 
Sanders and Dfean H. Nafziger, Will "be ref erencfed *hefe. wherever relevant, 
to the pres^ent 'topic. • . ' 



'I' ■ 



. Audiences for/ thi s ' pa^ er. Guidelines' ariffi *sSu^*gestibns on the 'topic 



^/pf'chis pAper are pot:ehtially useSul to max\^ and inditriduals . 

Etias::j^^er^ might be^ div-ided-'lnto ^Wo ^UdliSTfb%3': j'<iy^ind.iv,iauars * 
! ' and/or ag^uci&"5-.wh6 provide eS/aiuatidn ^erviqes/and .(2) 9liefifes who 

require such evaluation sehiices-^, Vn £be* pr'e^sent cade in A.-las.k'a,* the 

^ ' ^ ' '\ ' \ 'C^ * ' ^/ . ^ 

. ^pacific audliences for which t^s, paper wasXrequested to .be' prepared-. 
' . . . ' r- ' ' , 'i '"t ^, . *^ V 

weije specif i*ed' as tlie fgilo^in^: ^ ' \, 

' ' .1: - Personnel ?in tlie Ai^Ska* State DepaVtment who .have 

fesponsibriliCy'for 'furKfing of arranging for evaluation 
^ * * . «^ ' * , * • 

* * f " f i » * 

^ Studies (e^g:^^ personnel in the Alaska State Department 

of Educatioji Office \of Federal Programs) ^ • 

^2^ Personnel in' the' School sy^Cems, districts and individual 
^ ' " I schools who' employ the services of others to ^ evaluation 
^ ' • studies in th'eir schools (e.g., directors of. special 

' ' projects* funded thr^gh the Office of /federal Programs,' 

school superfntendknts, and the^like), and* 



3. Persons or agencies who currently provide or intend to 
provide in the futuij'e evaluation services to individiials 



or^ agencies in ^i'ther of the above audiences. 




./ 



./ 
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' ' ' ^ Readers -GHicye tihe- Paper ' 
" Jhe* toiiii of ^valukti6n .conDrab^ing is broad a^d m^ltifaceted^ and- > 
•the effort to treat it thoroughly and coi^Lprehensively^ has' resulted in . 
;a sizeable' (perhaps, too long) document organi^zed into S)&verai ^^^1^ 



sectibris^ -Betause of the length of th^ paper, two steps 'have be,eti taken, 
to aid th^ reader . - , ^ ^ >^ _ - ! * . , ' ' • * ' V ^ 

* First, each'of the majot sections "listed in- thfe tabid of contentjs ^^'j 
is''^nnotated below to provide adequate information *to the reader who ia 



( 



i,ntere-sft;<d in only some of the considerations ' discussed , 
\ Second, the first major section contains a self-contained abridgment 
ana\3ummary of the paper for the readet who (1) wishes ayi overview of the 
standards and procedures suggested in the paper ^sufficient to apply them, 
and (2) does not'filnd it necessary to delv^ into Che extensive rationale 
an^ .discussion which supports the sjiandards and .procednres^ whi^ 
, prooos'ed. • ' - n * ^ "^^'^ 



The remainder o| this paper is of^anized into the following s'ectiLons: 



are 



Summary ^ahd Overview "of Standards and Procedures *f or Eval uati on 




section contains: TlT^a ^rieif^^discussipn 
of the use of the proposed^ s^tandards and, procedures (e.g. , when 



should they be -'applied,, by^whom,, how,, and to^hat,?^).; <2) a 
^checklist proposed for use in applying the stan4cird^"^rtd ^ 
prqcedures; (3) instructions for use of the.'checkldst; and 

■ \ . . - ■ / - ' • • • 

f ' (4) a flowchart ^hich shows< the interrelationships and 
'sequence of majbr events * for -applying the 'standards and 
' procedures. Each of these tb^ics is- drafm from* a more '\ 
^ --epctensive presentation 9f the same* topic in subsequent . \ 
section^^ of- the paper. The purpo3e^for- this secf^ion^is ,to , 
'f, provide sufficient in£ormatiori fbr the person interested;^ 



/ 



only in the standards and procedures and tools for applying 
th^ and ribt in the rationale for theiy development. For 

anyone interested in the* rationale, tiiat is contained in 

' " / « ^ ^ " . 

the 'more ext^ffeive sections' ^hich comprise the remainder of 

this paper. / ' ' t , 

Rationale and Dijcussion Relevant to the Development of 

Standards and Procedures for, Evalaation Contracting . This ^ 

major section includes the 'ex;tended discussion ^and . rationale 

referred to above and therefore will be 'relevant* only to the 
I * • 

reader interested in the details underlying the standards • 

, C . . • / • > ^- 

and pracedures summarized in the first section. 'This pajor 
tsectio^ is subdivided into the ^-following sub-sections. 

A. Uses of ^Educational Ev^luatjLon . This section' contains . 

. r " ' 

a discussion of the general purpose, of evaluation- and 
- specific |>urposes ofc twq types of evaluatioti, formative 
' I and summatiye, ^and' wfio' might besV^^fonn each. 

B. ^ 'Co5[ce£tual^ 

Appropriate . . It^pakes littj.e sen^e to di'scuss settring 
• up an evaluatipla .i^ntract 'if conditionis, do not really* 

warrittt-^an.,,^^ rirst' place. Soirietimds . 

. evai^ation, contr*ac*ts-^are'^^s-tahJj;Sh^ which have ^ 

di'lcfUculty f roiQ the outset becai^e eValuaTtion activities 

are Unwarrailte/d . This seQtion is*an attempt to present^ 

ways O'f determining when evaluation is appropi;iate atiji. ^ 
, ' ' ^ ' • » - J 

. ' ' when it is not . • 



Rationale for U^se of External Evaluatlbj\ Contracts . This 

^ ' ' \[ ' 

section contains: (1) a discussion of inV^rnal ^and external 

\ ' ^ % ■ ' 

evaluation *^nd how th^ relate to sunimative\eva3>i4.ation, and 

(2) a d^iscussionvof the advantages and disadv^ntagesN^ 

contracting with external agehcies or indi^^iduals to conduc't 

a sunnnative evaluation stijdj^, 

* \ 

.Ratipnaie for Specifying Contractual Procedures . Not; everyone 
agrees that evaluation contracts are either necessary 6r - — 



useful. Howev'er, theire are many' probleiq,s in- the 'conduct of 
evaluation |tudies which would se^m to be'.best (if n^t 



ia^clusively) solved by estabj.ishing-,f inn' shared understandings 

\ . \ ' \ " 

andXagreements between .client and evaluator. • ThiS section 

^coTifea\ns the rationale for c^h^ use of/ evaluation contracts, , 

■ * > ■•/••■• o > 

Standaro^s and Procedures for /Selecting Evaluation Conttracto^s / 



It i*s oitan a large- step* from deciding! to contract with an external 
evaluator to finding a well-qualified evaluator t& undertake, 
the work. This section contains:' (1) ^a general dis'cussion' 
of problems in ident if ying\ competent /evaluators, (2) presentation 
of three possible approaches tp judging the qualifications c^f 
an individua!l evaluator* or staff of an evaluation agency, and*' 
fs) discussion of pro(^edural opd^ons-^^for selecting from among 
qulalif led ,evaluators • , ^ - , \ 

Standards and Procedures for Negotiation W^th an Evaluation 

/ r V — - ^ 

Contractor . In mdsjf ^ instahc^s^, establishing ^an evaluation 
contract requires sbm'e "negotiation or clarification between 



/ 



/ 



t 



'I. ' ■ . ■ 



evaluator and client. This section. 



with six relevant 
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topics: ', (1) when to initiate negotiatYons^^ (2) consideratiot 
in agreeing on an evaluation- design; (3)\ how tq plan the: ^ 
scope of work; (4) issues for discuss^.on and mutual agreement; 
^50 parties to be involved in, the negotiation; and (©^^ procedures 
for terminating negotiations. 



G. Standards and^ Procedures for Monitoring an Evaluation Contract . 

^ .-7— ; — — t 

Even after, an evaluation contract has been esqablishe'd , it is 
important for ''clients to monitor performance on the contract 
to* assure timely performance and to ^void misun^grstandings 
' or dissatisfaction which might grow out of even^i^'Xftitctreseen 
when the contract was initiated. This section" lists the 

. J, 

• basic tools for monitoring and briefly describes procedures 

' appropriate for use in modifying, re-negotiating or t;jermiW|4hg 

ft 

^ contracts. 

H. ' Applications of the Standards and Procedures to Samplfi Contracts , 

This section is presented in the hope that sample contracts 
may help illustrate ^.the points and issues contained tliroughout 
this paper. Three contracts have been prepared. The^first 
is a contract for the development of an evaluation dasign. ^ 
\ The second is a contract f or- Che imjilementation of that design. 
* Both of these contracts are comprehensive documents, written 

J e ^ — — 

in $i IctKcmal style and with ^ach party -s responsibilities 

explicil^P^detailed. The third- contract is a simplified* vei;siOn 

^ \ * \ , 

of \t;hef^conci contrac^V"^and represents a document which •woul^ 

V^^A \ ^ * , c • - 

servexas An>^dequate contract' in man^f, perhapr# most, circumstahces* 




11 
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I . fSUMMARY AND" OVERVIEW Of STANDARnS AND PROCEDURES 
FOR EVALUATION CONTRACTING 



/ 



When a client and/or evaiaatbr contemplate (s) enter iftg into a contract 
to evaluate an educational program or project, it can be prasumed, that 
several prior dec-ision? have been maUe. * 
' Fxcst, it evidently has been defcide^^^ that an ©traluat?ion of the program 
or projec^t is appropriate* Someone' apparenb^y has 'decided that: there are * ' 
relevant decisions to be made for which evaluat^e <iata would be important/ 
or that ^v^luative feedback to program staff would \elp. to improve the 
program* Or perhaps there is a legal requirement thatS^the program be ^ 

evaluated, whatever the rationale for ^he decision, an antent to evaluate 

* \ \ 
is patently a r\ecessary precursor to an eVa^uation contraf^n 

.Second, it evidently .has beeri oecided that th^ evaluation^ should be 

#f V ^ ' . 

conducted by an individual or agency external to the educationaAinstitu-^ 

tion responsible for the program or project. Someone apparently he 

realized \that q^ttain types- gf evaluation studi^ (je.g. ^ , end-of -project^ " 

\ 

evaluations designed* to tell funding agencies if ths 'school district 
accomplished what it set out to accomplish in. the project) ^re les^ 
'Suspect; if conducted by' external personnel. 'Concern for credibility. 
obj eo'tivity* of the evajGaatioi;i. results li*kely have pr^Dmpted ,jthe decision 
to seek extetnal assistance. Wha*t^er the motivation, contemplatl.on 
of ^an evaiuation contract would ^sejpa to su^ggest the notion ofr involvement 



tne upci 
Lstler ar 



of , an^ external, party, since only 'the most pessiiaistler are likely^to 
require'".coni;radts wh'&n the, evaluation is to be conducted by individuals- 
employed within' the . same institution. - 

• ■ ■ • 12 ■ ■ : . - 



r 



Third, it evidencly has been decided that the evaluation should be 
conducted undef a contract or 'som6 similar formalize'd agreement, rather 
than having the external evaiuator j^roceed to carry out the evaluation 
with guidance only 'from whatever info'i^ial verbal or written dialogue " 
may' have occurred. Someone apparently has decided that clear, written 
understandings of |he conditions under which the evaluation will be * 

carried out and of., the expectation;s of all^parties will help avoi>d(aater 

• ' ' . [ 

misunderstandings and disagreements .concerning the conduct or ^resmlts of 



;'^)aJjuat 



the evaluation. An administrator who has never yet received an e 



report In time for use for the decision for which* it was intended may 
wish to require delivery by a specified date as a requirement ^for fu^l 



ion 



payment' for the evaluator's services. Or, ^n ^valuator whose l^st 
evaluatibn report was rewritten by th^^iient (With remoya^Tor all 



negative findings) before , presentation to the fundiiTg agency may wish to 
stipulate \n a formal contract that no chaqges be made in the evaluation 
report without expr^ess '^'cojisent of the ;infthors. ^^f^^haps it has Veen 
concluded that, even among administrators and e^^luators^with the highest 

possible professional standards and ^ethics, conflicts can and do arise ' 

-* " , - ' ■ ' 

which~cbuld have been^Iargely ayoided had there be'en better documentation ^ 
of shared understandings concerning important procedures for completing 
the evaluation. 

The basic thesis of the later sections from which this condensation 
is drawn is that the use of evaluation contracts is advisable when, 
evaluations are to b^ conducted by persons external to the institution 



responsible for the pr6gi:am to be evaluated (o;:, in 1 

13 * 




institutio^? 
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*suclv as largj^ universities, by persons external to the unit o'r department 
xesponsible r^xr the program)*^* vThe rationale presented in the^latfer sections, 
has led the authors to propose a set of criteria to assist, administrators , 
and e^^aluators as they think about whether. to set up an evaluation -contract 
and, if so, how -to go about ^t. These criteria are summarized in this 
section, in the form of a checklist. A flowchart, in whj^ch the major 
points *of the checklist are translated into a pictorial sequence of events • 
and decisions is also included. " " ^, 

A« Users Guide to ^he Che'cklist * • 

Before presenting the checklist, it may be heljiJtiil to discuss briefly 

its js true tare, who might use it, ,wlf6n, and for what. • ^ * 

% ^ ' ' ' * ' 

First, the checklist contains seven sub-sections which deal respec- 

^ tiiveiy with the, following types of critferia: . , - / 

1. Criteria for det^rminihg when to conduct an^ evalual:-ipn \ 

2^ Criteria for determining whether to 'contract with dn ' . 
ex^ternal' contractor 

> , 3. Criteria , to consider when selecting an evaluatdr 

4. Criteria for selectJLng^ among procedural options for 

Vletting a contract ' ' ' ' ' 

' * 5. Criteria to consider when using a Request^ for Proposals 

* ^ 6. Criteria for use in negotiating the contract 

7. Critett^T^or use in monitoring the contract.* 

V Different sections of the checklist will be useful to different 

individuals' and ''groups for different purposes. *^ong the more * fibvious 

users would be those listed belowy 

- \ ■ ■ 

■ . 14 ' - . 
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. Funding ageWy^ tfersronn^l .(e>g> , personnel in a State Department df^ 
•Education Of£ice pf Federal Programs) ^would likely find all seven sections 

f the checklist useful as .they discharge their comprenensive responsi- 
' bUliti^s. • Section .one could serve to assist in decisions about which & 
pnograms would mo^t profit f.rom or require ey^l<fation. Use of section 
twif^ should result 'in 4 d^eci'sioh. about whether* evaluation shouldy be 



ting tHe 



iucced internally or externally to "the. agency conducting tHe programs > 
k)nS' three though seven would b'e useful in, cases where funding* t 
rsonneJcbecotne involved inr esta'&lishingi^ the evaluation contracts 
direcciy,a^=^ell as,.cases where they assist edjicational administrators to 
^ arrange evaluations mandated under titled funding \programs (e.^^ , ESEA , 
^^ici.e-^I) but left to t^e discr:etion of the distruct to arrange. 

' "^""^'" ^EducTt^^ (e.g., '^chool su^l^intendent or principal, 

\ ~ ' ' ' ^\ * * 

'dean of a college of education) would find all^sfedtions of the checklist 



el to-t 



useful in a manner parallel to -th^g -described abova for. funding agency 
personnel (with Che- exception of the pro^^iso relatiq^ to' assisting 
educational administrators).. In short, whenever k program or project' is 
^ ^ * in need of a summative evaluatioit, the entire checklist should prove 

■ - V ■ V 

■ ' useful to an administrator responsible for identifying th^ h^st evaluation 

> / \ ^ ' ' ♦ \ ' ^ • 

help possible^ to carry out theXevaluatiou. ^ 

Educational evaluators would^.have some interest in s6(^ions otie'^'two,* 
and^^fpur and would provide the data on which* judgments in section threfi 
would made. However, active participation and use of the -checklist 
by the evaluation contractor would involVe sections five' through seven, 
which bear directly on planning 'and conducting the evaluation. 
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Hoy to use the checklist . The, checkli^p is largely self-explanafiory . 
Users would notmally proceed t,hrough the relevant sectip.n(s) of th^ 
checklist, checking a single respons,e optica :or each criterion lisC^d'. 
At the end of . each, s.ection there "is a suimnaTy question which^ sholfld txe 
answered on the ^basis of the responses-'to the previous questions in' t|iat 
section. It sh6uld be noted that ^o .fbnauia^ or \rules ace pr.oylde4* f or 
how ope draws 'the, sumjnary judgments for each section of the ^dh^ed^li^ t^, * 
The checklist ^is proppse^ as ati.aidifeo, and not a- substitute »f or , ^ *\ , ^ 
orofessiorial judgment. In the autff6ts'v'''judgment attempted rijles\or, * 
argoritfims f6r summar'izfhg criteria *f<)r-eaeh ^sect;ion 'wotild result; in a . ^ 
mechaAist^^id approach that would ill\ serve the purpose- for which it-is^*'-^ 
intended. . Each»u*ser must acc;ept responsibility ^or his^ or/her own * *. 

Mudgiaent. This caveat notwithstanding, iij ca^s whdte several ^ persons ^ 

' . ^ r ^ ^ - ' ^ ' . ' ' . • ' 

'apply the checklist to cotmnon entities,' it may.be important ta agree *on^ ' , 

'common interpf etati^^^nd weighting among criteria/b'efqre- ^ggregat^ijg ^ 

ii^dividua>l judgment^ltfdsSi. raters. * ^ ^ '/•'^ ^ - ' , - 

'Two other factai's, should also be 'noted. 'First,' it is not.^assutned 

»that aXl users will have at *thet£ 'f-inger^tips aJUL' the 'information X^ecessary 

to ^swer .each .question"' in the checklist "(e*'.!* , que^tibns d-e^ling with 

the*adequ^cy ' 6f the jnethodologicar backgroiidd .0^ a'-prpapec^^ive evaluator); 

It hoped that those whtf need.^assis trance will turn to othejrs with I 

' . ' . * * . * ^ \ '.^^v * • . 

, known ^expertise in^^levant at^elds ^Ifiieri qu'est^ons, requi'te* knowledg^'^^ ♦ 

qe/ond thVt 'ppssessed'^y the us^r. -Xf .siich heip .is unavailable^- .th§ 

question may simply have *to;.gd unanswered.' The^yiiask-'of * Uhclyding^ some 

anaestions whiah' canpot always b^e- answered unequivoc^rliy seemS/ lasts', ' ' . 



^however, than chat of otai^tlnlj the consideration from the checklist ^ 
'simoly l^^cause obtaining an answer to Xh^'t^ question is sometimes V 
dirrieult. * y 

* Second, th^ several sections of ahe •cJieckli^C'^eal w.ith different 
consid'erations , making it Inapprppriate'to reduce all sections to a' 
completely co'nsis't^nt format. In some sections, one item of ne^cessicy 
lead^s to another in strict sequence, whereas other sect^ioils .contain • 
prof ij.es .of ^i terns which could .be easily answered in orders opher than 

.' * *^ • ^ i ' ' . ' , 

chat in -which they are/ presented . 'These format changes should not , 
detract ftom the cl^ecklist' if prospective users remind^ themselves -that 

the criteria a^e intended to prompt thought '2li>ou.tp* important aspe^-ts of 

*, ^ ' , ' ' 

■* » 

contracting * for evalaiatjions and facilitate ^decisions about that 'process 
and'.'^re not .intended to produce a "score," " . - 



B. Checklist; of "criteria ,for^^'aluatlon Contracts ' " 

The tomolete Qheckldst is presented below/ followed by the , translation 

of the checklist into a flowchart of major/events and decisions relating . 
. ' ' ^ ' * ' ?^ • ' ^ \ , , ' ' * ' 

to the criteria pr6sen|y|d in* th^e checklist. ' • • * 



* CHECKLIST Section One; Critema for Determining 
/ ~ ^ • When & Conduct an Evaluation . ' ^ 



1. I« there a legal requirenient to ^evaluate * 
the program? (If yes, initiate. the 
evaluation; if no;' go to Item 2 below. ) 



Is there a decision to be made for 
which ^e'vaiuation information would be . 
relevant? (If yes. go. to Item 3; if no, 
evaluation *is inappropriafte and you*shi)uld 
di^cooftnue fui^ther use of ^is checklist. ) 



Is it likely , that the evaluation will provide 
dependable- information relevant to the * 
decision to be made? (If yes, go t<J*^!ftem 
4\iftio, di^confilnue. > * - . * • ^ 



Will ^he 'decision be made exdusively on 
otjaer bases and uxrinflueaced by fhe , . 
evaluation data? (If^yes^, /evalua.tion is , 
superfluous, discontinue; if no, initiate 
.evaluation. ) ^ ^ ' , /j . 



Check one for each it^m 



Yes \ 



No 



• . ' '' ^ ^ * * 5 
SUMMARY TO SECTION ONE: : y ^ \ 

Based on questions i-4'abdVe, should an - .j 

evaluatipn be cotidudtecl? (If.yes,, proc^d - ' 

to^qhecklist section 2; if no/ discontinue / , 

use of this ch[eckHst.') ■ * ' . 
* * * ^ i — ' 






> 




Ml % 



t 



CHECKLIST; ^ Section Two: Criteria for Determining: Whether to 
Contract With an External E Valuator 





Check one for each item 


Yes ' 








1. ' Is there a legal' requirement that the evaluation 
. be bbnducted by an outside agency? (If yes, 
^' * initiate the coVtract; if noTgo^to Item 2, ) - \ 

# 




/ 


**> ^ MA *»ae/MiT*rtae OXTOlloT^lo cnrM'v^T*^ * 

^, Are tinanciai resources civciiiclijic uj aupputt ^ j 
' a contract for an external evaluation? (If 
yes/proce^ to Item/3; if no, ^c^nduct the ' , 
evaluation internally and discontinue usfe • 
of this^checklist. ) 




i 


r , * • ' 




/ 

1 - ' 
* 


* * 

3. ' A^e credibility and objectivity of concern ^ ^ 
to the audiences far whicH tfie evaluation is ^ 
conducted^ -<H yes, pr9ceed'to Item 4; if ' ^ ^"^^ • ^ 
no, discontinue.) ' \ 1 f 


i, . 

% 


f 


J. Tc fV^prp Jin pvfpmaT. Contractor Dossessinsr • 

' the necessary/teohnical competence who is » 
available and willing to do the. evaluation?* 
(If yes, initiate the contract; if^no, conduct ' \ ' 
the study internally, but recognize the loss 
of credibility axid objectiyitjr. )^ ' 




\ ' > 't 

5 








SUMMARY TO SECTION TWO: ' . , , 






T^flRA^ on questions -1-4 above, should this ' - . 

evaluation be contracted with an external ^ / 


I 




evaluator?' (If ^ y es, pfoceed to section three ' 
of'the checklist; if ho, discootinue its use. 


^ 





■'*Further detail on this criterion wH|be providrd,,iu a*subsequent secti^n'o^ thls cji^cklist 



1 



\ 




CHECKLIST: Section Three: GTiteria to Co nsi'der Wlien . 
, Selecting an Evaliiator ,! - - ^ 

(Each, item Ijelow is written to apply to ai>,individuaL 'if Ihe pbt4ntlal contr^tor is an ^'^ , 
agency the (^uestion^should be re-castraccprdin^ly,)*' / ' * \ ' ' \ 



To wiat extent does thel formal travains: 6f 
the p:>tential contractor qualify-^him/her to 
cbndiict evaluation studies? (Consider 
itemi such as. major or minotf degree 
specialization; specific courses or course 
sequences in'evaluation methodology; 
qualincations of persons with whom 
'training was taken, ♦ if such qualifications 
are )<nown. ) 



To what extent does the previous evaluation 
experience of the potential contractor^ 
9i^*alify him/her to conduct -evaluation ' 
studies? (Consider items such as length 
of experience; relevance of expert e nce.js _ 



CJontractor applkrs to be: 
(Check one for each item) 



Well 
Qualified 



Cannot 
Determine 
Qualificatious 



Not. 
Weir 
^Qualified 



3, To* what extent 'does the previous. perform ance 
of the potential evaluation contractor qualify 
him/her to conduct evaluation studies? 
(As jlidged% work samples-or ref^renWs). . 



' ERLC 
1 



SIIMMARY'TO SECllON THREE; 



Based on questions 1-3 above; to what extent is 
the potential evaluation contractor qualified to 



dbnduct evaluation studies? If "well qualified. 



proceed to subsequent section's of this checklist. 
If '*not well qualified, " discontinue consideration 
of this contractor, 'if ^'cannot determine qualifi- 
cations" on basis of initial information, seek I 
additional information and/cr assistance from 
colleagues with expertise in 
this critical judgment 



evBfluation to make 





•NOTE: A neutral category.has been included ij 
^initially available about potential contractors 
^judgrnents to be made on this dimension. 



tkls section of the checklist because informatlc 
ay^be insufficient to permit unequivocal 

1 ■ 



1 



*JL7 
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CHECKLIST! Action Four: Criteria for Selecting: Among 
Troeedural .Ofitions for Letting Contracts w 



Check -one for each iteiri 




Is there a legal constraint against $ole sourae 
contracting?- (If .yes, discontinue u$,e of this 
section of the checklist and go^to Item 4 below; 
if no. go On to Item 2> ) 



Yes 



. No 



I& there an evaluation contractor who meets \^ ,^ 
'the criteria in checklist thre^, ^ who is . \ 
iiiterested in this evaluation,- and whose 
SeVvices are desired by the client?^ (If .yes, 
eo k> Item 3; if no, go to Item 4, ) 



vw ^.^^ ^ ^ ; : ! 

3. Have such a large number of contracts been 
let to this .same cotifaractor that "cronyism" 
may be a danger?/ (If yes, ^ reconsider; Item 2 
above for a new contractor;, if no begiii 
negotiations > ) 




Are 'the financial re^o^rce^available for 
\this jevaluation sufficient •to warrant the cost 
of atti.RFP? (If yes, prep^rfe an RFP; if no 
usie synopsis review approach to *fin,d a ; 
stU table contractor. ) ^ ^ ♦ 



gUMIVIARY TO section; FOUR: * 



Based omquesfions* 1-4 atove/ which of t^ie 
following procedurar options do you select 
for letting the eValuatioa cQntract ?> (Btiet 



defmitions of option^ appear belo^^ on.this'^ 
page.) / u \ • v? , - ' ^ ' 

(If "a'^^oXc'^b*^ Is phoseiji/go to section l^ix of 
thi$; checklist.; if\TD" ts ch6sen, go^d section 
Sve o/ this checklist ) ' > 



a^: 'trn ^ole Source Contract ^ 

b. ./Q| Request .for nrbposajs' 

c. Qj Syndpsis Bevxew Approach 



4 



<^le Source = contractor i,den.tified solely on ;tb6 basis of Known quaiUfidatibn to conduct " 
' ■ ' evaluation studies- ■. • . V, 



* Request for Proposals - cpntractor identified on iQi^'basis of a detailed bropgsal f5r the 
•|- . ' .conduit of tiip evaiuatio^^. subthitted in an open cd.iipetitio^ for .the f onfr^b^! 

skLpMs Review Approach = contractor identified in the l>^is of dbrilf outline oijl) h|i 
. ► /T : the evaluation study wouAd l)i3 iondutted /and'"(2) evalvltorlfs .qualifications- \\ 



CHECK-LIST: -Section 'Five: Crlterfa to.Cdnsider When 
..I -~ ' " Using an REP • , , ^ ' '• 



\ ' • 



1. /Doe§ the RFP- include: *' ' ' • ' \ 

' ' . . . . ' . ^ \ ' ' ' 

a. an adequate description of the context in • ; 
which thfe Jvalupfti6tt!is to'tak^ place?'' 


* * 

♦ 0 
• 


.Check- one for each item 


Yes 


No 


if «. 

* ' ■ r 


• 


b. clean st4tements of the^: purposes and^ \. . ** 
. ^ • . characteristic's the project to be^^ ! 
evaluated? ^ • \ * ,c '.^ " - * 




■ I 


c. * the* specifi^J ptVpducts e^xpeeted fSsm-^ ' - . . 
*. the evaluation? ;• ^ , 






• d,^ the ^.eisources available for the ' , . 
evaluation^ ,r , ^ ^ « 






* • 


^ e. the copstr^ints -which would affect the .5 ' • ^ • . , 
' evaljiatfon? * ■ • " ^ j , * ' ' , ^ . 


< - 




'-^ — ; ' ■ — ' ■ . — '■ — r r: — ■ — 

.2. If methodological restrictions (e. g.,, ' . ' . ' 
• • restriptidns -on sampling) are essential, ■ , J , " ' ' - 
are they include^ ia'th6 R'FP?\ . ■ / 


\ 

* " m n 
'/ ' ^ 


1 

. / 


3v . Has a*pan?l been selected for ^eviewitlg , * u 
thS proposals received? ^ ^ \ 


V * • , 






— ' — — /\i . '^^r.:.^ — 1 

t4. \ : Has the panel established criteria for,^ • 

I ' judging, th^ proposals ? * . ''^ . ■ 5 ;^ \ 




» 




'5. ^ ^ Are tted^e criteria included in the RFR? 


1} r ■ 


^' 





SUMMARY 




4 , 



CtlON FlVE r 



Based 6n questions I'-rS^above/. is> the RFP - 

issue|Rffy-and 
checkllstt if 



ready to benssued ?c (If ye§ , 
proceecj to section s \jc pf the 



no^ .re-Write RTi^).. " 



,1 ' 



^2 : 




^ CHEC^LIST^ Section Six; Criteria for l/se iti ' ;^ 
' Nagotiafing the\Contract . ^ 



1\ ' If au e-^e'rixnental design is appropriate,.. 
' " / are th6 negbtiationVsche(3uled;to occur in 
time to pejfmif evaluator involvement prior 
to the ^issignment of me f r^eatmenj 'to thfe, 
Experimental units ?'V / 





v: 



2. time frai^e approach to ithe evaluation ^ 

is approplriatp, .are the negotiations 
\ Scheduled to'occur in time to permit the 
*\ \ collection of data (concermng the status 
• ^ of those who are to participate?) well in 

adyance^of ti^ intr.oduc1ioPj^%the treatment? 



^3. * * Iluecessary, are the negotiation^ Scheduled 
- ,to occur in time to ^rmit Hie development/ 
' ' \\ tryput, .analysis,, andp revision of instrupaents^ 
prior to"^ thfeir first intended use? 



4, ^ Have adequate provisions been njiade for , the ^ 
development of an evaluatioij design, through- 
' -J. . one of ^the.following: preparation of desi^ ^ 
\ • for 'set fee; preparation 5f design as ttie*. . 

'X^ ^ ^firstphasd of the coatjract^Wprlgpar^^^ 
, \^ ' of d^srign uti'der an initiaV'Cpntrkc^, wi^- ^ 



\ - > ' separate coatract for implemeuting the desig^. 
^ Poes the negotiatecl Scope 'of^T^ork includ^p: : 



a. the proceduV,es tcJ be emplbyea^by the, * 
* ' evaluatoV and a time ^ schedule fpr their 
performance?''-^ t - • Iw \ 



b. adequate descriptions of the pro^licts to 
be expected from. the evaluation >iid 
deadlines for theig deliveil)!? ; 



c. criteria fo?^ judging that tft^ conti-act 
' has been fulfilled ? ^' 



Have the negqtiations dealt ^ ^ti^the 
respective responsibilities of the client 
and.contr^actof coticemingt 

a. the ii^n^catic^n th 



•■/] 



b* data acces 



objectives of 



J[L_L 



A 
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Check 6ne for &ach item 



-Yes 




^No 




a: 



^1 
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Are the audiences pf Ijv? eVa:luation 
repQrt(s) identified^?. 



^w^M^w^w^ > . 

8. Have negotiations^ c:oacerning potential 
conflicts of interest, taken place ?/ 



Check; ong for each item 



Ye 



9. Has the issue 9t/'piggybacking*'yb£her 
research intei»e.sts-b;gen dlgcuSySed? 



10.. Have the copy?i'^hf:^iiS publics^^tion 
rights issues beVn iioroughly,* ' 
^ explored and i^u-Wllv- agreed upon7 



11. Has an o^^hemeni copcernmg the 



^ r'ights.of priar 'r6,yiew of reports^ beeu 
"reached ?. * 




His an adjudication procedure be^n 
^ ^stahlished? 



Has- a basis for payment been 
established? 



14.' ' ,Have the primary i^pgotiators l&een 
* identified ? 



15, Have those who will be affected. by the 
\ .evaluation be^n Invited to particiQjite' 
at a, secondary level in the negotiations, 
at leas t by representation ' 



16. * IHas k tinje and'resburpe limit for \' • 

negotiVtions been^set Id^ mutual agreentent? 
i If '^e negotiations have Hot been success- 
) /fiiUy coWudedwith thle time and resciuMe 
^ limits set,- has a i^econd potential , '''"^'v 
contr^cttJr been identified? -^(R^er to . 
t\aec 




checlAlist \^ecti<5tx thi:fee^ ) 




SUIvtMARY^ W sicTlbN SIXr 



lave lie 



Based on queL .r^^-- M^t^—^ 

been sufficleiMy auccessfujl to vrsjrrantj Utting^al . 



tidns *H17 abovej 



contra ct or^ci^oHtracts for jccompllishml 



evaluation dVsi^ and, conduct of -ihe evaluation^ 



(lf ye^\ sign^tlie^co^iract; If no,, returu* to sec^i 
four ind yse to|find anoliier contrac^K)^ 




CHECKLIST: Section Seven:' Criteria for Use * 
' Monitoring th^XDofatract ' . \ 



Have procedures for pvonitoring the progress of 
the work .been agreed ilpbn, i-. e. / hav^ 
milestone 'feeView points and jifqduct.d^adlines 
been clearly set? ^ ' . \ ^ , 



2. If"6ircun}*stahces are different thatn ftose ♦ ^ ,^ 

anticipated^ in the-^ntfect, has a'renegStiation ,^ . 
oc*curredJ5V> ' ^ * ' * ' • ' ^ 



3. ' If Vepegotiatian^was ndi possilDle,. has -a mutaally 
^ \ s|af isfac to ry agreement lo terminate \he contra<^t/ 
beeij- reached f~ * " c ^ 



\\ 4.* / If a, dispute exist&i^ have the two* parties met to . 
\ attempt a r.esolution? HaVe tha rps^ilts of that 
•meeting been dodumfented,?. 



' I If the ^dispute cannot be :pes,61ved among tljie 
[/ parties 4td' the contract, \iak th^ adJudicpti|Oii 
pr.ocess been invoked^ \^ 1 • ^ 



(In 



Chebk one for each iteln 



\ 



No 



i 



If 



'eric 



S UMMARY "TQ SECTION jSJIVEN : 

* " f V or \ A 



''leased on questions 1^^5 above r ts the conduct of the 
evaluation prqcdedlng in \a mutually satisfactory ~ 



fashion ? (If ye^ ^proc^e^/to completion ofihe 



evaluattoa;'if n6, modify or terminate contract 



as 



nj^essary^/) 



















s ^ 


1 

/ / 


1 
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C.' A Flowchart of ^EventV \ ^ * ' ' * " ' 

' ' V- \ • ^ ^ * ' ' ' . 

In th*e previous section\ of tiiis paper, ,the complete checklist 

• ■••••* i \ * • ^ . . ' • 

relating to contract standard^ and pr>ocexlui:es was pi^esent^a. In the 
-% ' , , \' 

. ^ 6 ' • 

following pages, chat checklist has been transformed into a flowchart 

to* clarif,y the t iking and sequence of the events in the preparation 

of ail evaluation contract, ^ ' 

The symbols used in this flowchart arenas follows: 



o 



/ ' 



I I Rectangle — an event ' ' \ 

<QJ)> ' Diaiti(|)nd — a decision. poiiit ^ ^ ^ ' 

' i' . ' * " 

' (2) Oval ^~ a termination, pbint for »goi^ thrpugh the flowchart 



\Circ]|e — *a conninuatio^n or transfer pointf 
^^^^ Hexagjpn y- end vpSroduct desired 






"I 

. •/' 
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/■ RATIONALE AND DISCUSSION^"~Se:K:A^^ TO THE\I)EVEL0PMENT' OF. 

^STANDAkDS AND,PROCiEDURE?^FOR EVALUi^TION CONTRACTING " 



_ The "^remainder (and bulk)- of this, pappr/contains : (.1) a;aore complete, 
presetitation/pf the standards ^nd propedur^s- prfesentfed in the previous 
section^ and (2)' ,.fhe extknsive ra^ti'onale iand d^iscuSsion which supports 
^.^.thc^e standards and -procedures Althougli 4:he previous -section may suffice' 
for many. users who require only an overview of the material sufficient to 
allow them to .appl-y it,, other users may wisl\ 'more detail. This would be 

N 

: true for' persQils .wKo wi'sh (1) to attain -more understanding of the 
■ poncepts implicit in'the standards -and procedures, or (2) to, probe the 

. . ■■ V . ■- • • . ■ * ■ / 

rational^ on which they staijd. The;, remainder of this paper is provided 



Lsf\ i 



to satisfy both of these i^eeds-. 



V 



■ A*. Uses of .jiducatiohal Evalu^tiion 



Before, considering the issae of when to evaluate, it may be, useful 



vo consider the prior question of whetheriit is e^r appropriate to evaluate 
The need for,>; evaluation has so often taken the fon\ of .an unreasoned 
assumption or a strident asser:tipn that|' it is^possi)le to<fose track of 



. the-i?eal logic underlying the initiatioj;i of evaluat 



o.n studies.. The 



rationale for evaluation* has beeri ai^gti^d at some^ length in previous 
semitial -writings on evijluation (e.g^.'i ^Str'Wln,-^ 1967;' Stake, '1967; and 

n^re,. ' It may be useful,^ 
e basic purposes of'' 



Stuff lebeam, 1968) and [beed^not be repea'^i^^hp: 
however,, to preseri^t a\brief orientatioi^ tc 



conducting evaluation studi^es and som.^ simple|]de:^nitions of how tenns,^ 
'Will be used in thi\s paper. '* • ^ 



'olc (3)'^^ decisions ab'but whether to conCfilu^..6r terminate a program^ . ^^ 
.Tl\ese, considerations lead to differentiation betweei;i two' types of 

evalii^'tiior^: '^ormtJLve*. and summative; forpative eva'luation simpl^j^., 

' \ \ * ' ' ^ . ; 'XL 

teiers td eyaluat^ltfn 'that is cpnducted during ^hie operation^qf a program 

• ■'V ' ■' \ - • ■ ' ■ ^ ' \ 

.for tWe' express. purpose, or providing evaluative infornjat ion to program \ ^ 

directof* Vor' cftei'r \use -in' improving it. Tor' 'example,- during the deVelop- 

•ment of 4 new curri(fuiuin; formative evaluation could i^iyolve content 

ins^edtlon- by ^per.ts, pilot 'tests v^i'th small numbers of children, field 

tests with larger numbers ^of cKildren'and teachers- in -several schools, 
i . \. 

* » ' ' ^ • . \ \'. ' ' ' ' ' ■ i 
^and so forth/. Each of ^ these; -steps woCld result in immedi-ate feedback 

'to the developers vAio woiflri^-use the inf ormation' to'^make necessary ^ 
i • ^ • " \ • < 

revisiotis ^in the materl^als-. In formative evaluation, che' audience for 

t ' • \ ' ' ♦ 

* the evalu^tiod 'repor*t? comprises persbnnel in the prograip — in our example, 
. ^' < . 

tUose- who were responsible for developing 'the c\irriculum\. ' \Fprmative • - 

• « • \ • ^ \ 

' evaluatib'ti leads to (or should lead to) decisions about program develop- 

ment; (including^mod^if ication,. revision, /arid the like). 

. Summ'^tive evaluation is evaluation conducted for'the express purpose 

* 'of-judglng the Worth or -eff ectiveness -of that program ^for potential users^ - 
'rl"^. ' for whom it -has been d^eloped,; after the' development is completed. ^For 

^ example, after the^curricul}lm.]j^ck^ge iff^^cpmpleCely d^veloped^ a summative 

^ * ' . ' ' ' « . ' ^ ' ^ ' « . / 

" . Ivaiuation might ^be conductecf" to determine hg^ effective the package is 

\ \ with aidaCionai' 'ffto^ple of \ypical' siftodls, teachers, ancj s'tudents^ jat the. 




' j leifSjjT^r which it was ^developed '*Tn summative evaluatj^on, the audiences" 

£br the;|fevaiuatdpn .report 'include the potential users (students, teachers. 



' \ alhd''^ottt^«1)rdfQ^'siona'ls) and the'srurce of' funding (taxpayer or funding 
..^e ^g\eacy)V well ,as program personnel. S'ummat;ive evaluation leads to 
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dec 



:isions^ concerning K^etker programs" should 'be cfontinued /^rminated, 

^. vexpjandad, etc. , , ^ . ♦ , \ * ' 

-7^ \ Ifshouldtbe apparent* tr^^m-this' differentiation that* formative and^^ 

sumin;ative ^evaluation at^^^both essential to,' sensible decision making , about. 
^ ' • ' ' ' # ' * - * ' \ 

. any adi^catiqnal prpgram, since ^ decision^ need to be^ made durfng the '\ 

\ ' ^ \ . ' ^ 1 ^ . 

•deVielopmental sta^e of a RfpgrW-to imprOvfe, helj^, and strengthen "it and, 
at;,'}the' end of one (or more) x:ycle of^^tlie program to judge its finals , 



'worth and impact ^hd determine its •tfjitut'e.;^ Unfortunately , f?ir, too many' 
'educators overlook ^the import^ahce of formative evaluation and focus only 

• oa-a' summative eva^^uation at the end of the prbgram^-^ugh an^ ov^ej^iJjgl^^ 
is myopic, since^ formative evaluation d%ta collected' eailyff&n >ielp re-ciha'nne' 

, a program into more profiuctive directions $0 that time, money ^nd all typ^e^*^^f 

_ > ^ } . ' ' ' • * ^ 

of human and matet^ial rtesources have not been expended pn^the program in 

* • * 

.vain. It is little help to ititrolduce evaluation only when the project 

^ has reached or rjjearlj? reached completion. It may simply be too late* 
to .save It. . • » ,,jy^' ' 

Despite the imp*or.tance of formative evaluation, it wMl ibQt,, be 
discussed^further irf^thls paper**: That i^^ot ^bacaifse it la^ks ia 

^ importance ^(f or 3*3 noted above, good"' ft?fitt&ti^#'i^^O^||k'tr±on may >e in ' . - 

many ways as i^rucial or even more crucial than Summative ^evaluation tq/>, ; 

' • ' ' ^ . . , . ' 

the ov^^^i success of a program) , but Uecause formative, evaluation isV 

seldom contracted, for Uy outside agencies or inHiyiduals. Typically, 

♦ 

f^?rmative evaluation studies which have b^en conducted in Alaska have, ^ \ 

been p§rfor6ed by stafr members of the fjrogr.am involved — i.e., they . ^ 

• - - * S ^ 

have been internal evaluations. .Evaluation contracts are typically made 
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witb persons or agencies external to the' program, and in, almost ,every 
^instance these ^evaluations have been for suidmative evaluation activities. 



5 



Therefore, the fopus of this paper ,is ejfc^usively on summative^ evaluations 
contracted to 'Out^ide agencies or individuals. The term evaluatdon 
shall be used in the retnainder: of this paper in t-he circumscribed sens^ 
of rafe-rring only'to summative evaluat'ioji unless specifically noted 
.otherwise.. 



B.' Conceptual Issues in Ifespermining When Evaluation is Appr'opriate 
- The discussion xn-'tlie preceding section^ provides a framework for 



consid-ering when it j[s appropriate* fo\ an agency (e.g., a sdhool district) 
to initiate an evaluation 9f one* of its pt;ograms. XJhe' idea that evaluation 
is aiwaiys appropriate -for any program may be\ compelling. ±n an idealistic v - * 
•sense, but it ignotes many practical realities .\ For example, ia some 
programs, the decisions have already b.een made forv reasons other than 
.evaluative data. Cot^slder Jthe case where a program bas sufficient political 

\/ ' * ^ ' ' ' " A ^ ' , ^ . • 

^ppeal with important conatrtyencies that -administrators are clear about 
tiheir intentidn^to continue it regardless of what any evaluation study 
•^s^^wed about the ^program, even if it demonstrated the program was completely 



" • ^An argument for external formative evaluation has been made* 
previously (Worthen, 1974) ^and the present authors support the contention 
that this is* an importaat activity. •Kawever, few educational 'agencies 

, have , yet contra^oted for extter.nal fomlative evaluations of tlieir ^i)rjDgrams . 
It ther'fefore seems prudent to '.restrict attention in this paper ^to- the 
mor^ typical external summative^ evaluatioy. It is beyond khe s^ope of 
this paper^ to develop guidelines^ fori an activity Which is^^as yet largely 

• untried . , s ' 



• * ,t ' 6 ' 

worthless o<i criteria such as student achievement. In this* case, 

. evaluation can play no role unless it is to provide a justification 

. <<a:* 'cahciRuat^on of the {^og^am, a whitewash function that prostitutes 

the role of evaluation. So it seems apj^^opriate to conclude that 

evaluation should not be conducted when the decision will be made on 

♦ other bases a-nd will not be affeC;^^,by th.e ^eJraluatti jK^ti ditaj^^. -'-"^ 

A second situation i*S one. in ;which 'there is no relevant scheduled 

-^decision for whicK evaluatjion 'information is needed. 'There may be some 

. merit in a "dec*isioii-f r^re ^Jader 's Raid^ers'* type of evaluator 'capability 



' in the field ^f^ttucation, ^ut, given the ^ll^it}^of_evaluation resources 
(b^h, financial and humaii)_jand— tha--^i^ra€ _ 



i d f^r evalugiLiotr mtorma- 
tion where -decisions are known, it seems a questionable investment at 
the present. Therefore, it seems reasonable to question the- utility . • 



of condOcting an. evali^adion where there is no imfnfidiate decision which 



serve.d-.by »the irtformation prpduced..by the study. 




A' related 



*is where a-^tlecisiorf can be identif iei 



atid - 



inf ormat:^on 



needs relevant to* tha 



ion specif ie^v-liirt the likelihood' is . 



'extremely sma 



11 ^^^^ evalua^iqtt ^udy could produce, relevant if^^ormation. 



tor example, there may be an upcQm^^!g^4.ecl^$ion about whether to". continue 
a dropoiit prevention program forj which inf ormatldn *about th^ ef^^ts 
or the program^ jon dropout rates, graduation percentages, and-the like, 
would be very relevant. Unfortunately, thQ program only started one 



Brickell (1975) has-^provided a deligUtful example of a situation in i 
which hiring of minorities as teacher aides in^^a- large city school system 
carried such potent political appeal that even 'ijiad the evaluation findings 
been negative, they wpuld almost certainly have been ignored by the 
school administrator.?, ^---l--^''^'*^"^ i * / ' ^ * 

41 ' ^' ' • . • 



month before the decision is to be m^de by the school-board. The. 
probability of obtaining depertdable* infortaat ion 'about the effectiveness 
of the program (even in predictive sense) in that length of time is - 
so flight that it would seem wiser to spend one's energies^ convincing the 
school board that the decision should be delayed. 

»Che foJ.lo'wing is proposed ds 'a checklist for. use in dec-idi^ng* when 
it is- appropriate £o initiarte an eyaluatiiOn. ofv fh^ program. * * © 



CHECKLIST: gSection OneV Criteria for Determining 
> When to Conduct 'an Evaluation 



"-Is there a legal requirement to evaluate;* 
the program ? ^yes, jnitiate th^ ' 
evaluation; if no, 'go to Item 2 belovf. ) 



2; Js there .a decision 

Which evaluation information would be^ 
relevant? tif yes', go to Item^S; if no. 



is the re .a declsioffto be macfFfbf ^ ¥ 



evaluation is inappropriate and you should - 
discontinue further use of this checklist )^ 



Is/ft lil^ely'that the evaluation will provide 
dependable i^nformation relevant to the 
decision to be'made ? (If yea, go to Item 
4; if no; discontinue. ) . t ' . 



4. 



V 



Wiirtha decision be made exclusively on' 
other' jpases and uninfluenced by the 
evalu^ation data?* (If yes, evaluation ik 
supe*rfIuous^^ discontinue;, if no, initiate 
evaloktion. ) 



SUMMARY TO SE 




Chebk^one for each item 


Yes: 


No- 


t 


> 


■■^ 






■ • / 


)■ 




/ 


\ 




ft 

^ - * V 


% 

c 

• * - ' 

■» 



;tion one< 



c>fequestion5 1-4 above, sliouTd^ n 
ev aluattoaUfe c^d^ted ? (If yes, proceed . 
checklist secSStrS-fJXno, discontinue 



A' 



V 



C. Rationale for Use of External Evaluatiort Contracts 

There are m^ny possible arrangements for getting summative evaluation 

activities done, including the -use of internal program staff, external 

'/ • . - ' % • 

consultants or external contractors^ However, the use of an-^internal 

• V / ^ ^ / * . ' ' . 

staff member, t^^on(iuht a summative evaluation becomes problematic for 

reasons which mi^nt best be understood in ^he context of a fuller discjussion 

of the differences in internal and external evaluation. 

.The tem/s internal anfi external evaluation are largely self-explanatory 

ahd\ refer to whether the evalu^tor is internal (i!e., an employee of) 



or exteri^^tsao fehe program being evaltiated. A Title III program, might 

•. ■ - ' • ^ • 

be evaluated by an e(valuator who is a member of .the proj ^ct staff (internal) 

^ or by an outside agency working under cpntract with the State Department ^ 

• ' ' , ♦ ' ... 

of Education Cexternal).. There are obvious advantages and disadvantages X 

)f these toles. The ilnternal staf^evaluator xs 



"^with both of 



zs almost / 



\ 



certain .to know more about the pro^ct than is possible fo^any outsider, 
but he or she may also bfe so close to the project that complete objectivity 

' / ' • ■ ' 

;in viewing it is impossible. There is seldom as much reason to question 

/ 

the objectivity of the external evaluator (unless he or she is^ found* to 
jhave a particular ax- to grind) and this ^dispassionate perspective is 
erhaps'the greatest asset of being unaffiliated with the program being 
valuated, ' Cd/nversely, it is difficult for an external evalu^or to 
^ver learn as( much about the pjoje^ct as the Insider knows./ Note that 
the phrase %s much " refers only to quantity, not quality/. One often 
finds an irit^panal evaluator who is ful%^^ unimportant details about 
the project ijul overlooks several critical variables. Tf these bits of . . 
*'-'X'5.. ' ^ ■ . ' .. 



key information are picked up by the external evaluator , as Is sometimes 
rh^ case, he or she may end up knowing much "less overall about the 
project but knowing much more or importanM^ 

Formative evaluation is typically conduced by an internal evaluator. 
Knowledge of , the program is of great value here and possible lack of ^ 
objectivity is-not nearly the problem it would be in a 'siinimative evaluation^ 
Summative evaluations are typically (and probably best) co^nducted by 
, external ^y^luators. It is difficult, for .example, to knoy how much 
credibility to accord a partictilar publishing company s eva^luation which 
concludeq t-haC" their set of reading materials is far better * than its 



competitors. The role, of the internal summativei evaluator ils only 
infrequently appVopriate.r ^tiowev-er, in some instances- fihere is simply ^ . 
-no pa«silfility;af^obtai^ng- external help because of financial constraints - 
or absence of competent, personnel or agencfes willing t9 do the job. In. 



these cases^ the summative evaluation is weakened Iby Ci^e^ack^of outs.ide 
perspective, but, it might' be .possible to retaiji adequate objectivity and 
ctedibility by choasing the internal summative evaluator from among those'* 
who are some distance removed from the actual development of the program ' 

r 

or product' being evaluated. . • - 

1. Advantages of External Evaluations , ^ ' . * 

/ ' • ' ' ' ' " * 7 

A, The advantages of contracting with an external agenciy or ih4ividual' 

for- the summative evaluation* can ^bei summari^ecT^ as follows: 



.^nti£vidual consultartt-s^nd" external contracting 'agencies are not ^ 
treated separately in the remainder of this discussion since^ procedures 



for contracting would be similar in^cuth tases. 



1.1 \The external evaluation is more likely to be objective 

(i,e,, capable of s^jailar interpretation of data an(^ 
similar judgments by different but equally tompetent ^ t 
eyaluators) . * 

1.2 The external Evaluation is more likely to Ije crjedible 
(i.e., capable of being believed or trusted), ' 

1.3 ^ The external evaluation enables ^n agency .which initiates 

'V ■ 

an evaluation of-an ediiCational program to draw on evaluation 
^ expertise beypnd that possessed by persons within. the agency. 



^Many* school systems and other .educational agencies simply 

< • / • ^ 

. do not find, it feasible to hire sufficient numbers of 

evaluation specialists to, conduct the evaluations needed 
in the system but can obtain the necessary expertise through 
contracting with ^xternajT agencies or persons who possess 
such expertise. • , . ' * * ' 

In' addition to these reasons" tt^r Seefci-ng exter^nal evaluation assistance 



t^e use ,of an exterlftal e^fa^uacor is offen a legal requirement und^r 




mandates which legislatok^^'have laid down for many ;of the reasons cited 
eatlier, * * 



2. Disadvantages ot External Eviluatipns * " 

- ■• • • . i- 

/Although this paper supports the concept of external evaluation 
contracts, there are 'potential di^adyantages/of such arrangements which . 
must, be retognized and compensated fgr. Possible .(although not necessary) 



disadvantages include the following: * ' 



2*1 -It is' somecimes dif£jLcult to* ascertain, or be assuiied of 
- • the competence' and , expertise or the contractor in conducting 
evaluation studies {a problem 'dealt with lat^i; in this 

\ ■ \ • - • . 

paper>;j . \ \ . . • . 

2u2 ^ The extern&r evaluatc^r may be unfamiliar * with the phenoi^enonL 
being eyaluated, at .least at the ou^taet. • 

2^3 Feedback aifid \coinmunication may be less * immediate. fhar\ where 

• \ ^ " • . 

the evaluat6r is- within* the ptogram or^agency. \ 

^Tr^ — -Exte rnal Qvaluations are typically , mdr^ - expensive th a n ^ 

internal^ evaluations becuase *of the usual communication* and 
travel costs, overhead charges unavoidable to contracting 
agencies, and the'like. -(It is tempting to point out tl^at 
.attempts to economize should be considered carefully, however, 
if the trade-bff is to' sacrifice objectivity, credibility, 



and technical competence) 
None of these disadvantages is 



lit 



comp.elli:n^^n' itself , . but they should 
decision about 



be considered along with the, advantages in reac 
whether *to contract a- summit ive evaliation. In the\p.pinion of the authors, 



^V' unless* there are ^nusuaj^ 



than is usually the case* 
• . The following is pr 
not to contract with a 
evaluation • 





the decision should almost aXi 

circumstances which make one o\*more|of the oi^sadvantages more salient 



as a checklist for defe^iding whether or 
1 agency or individual to conducp the 



\ 



CHECKLI ST: ^Sgction.Two; Criteria for Pete nnining^ Whether to 
Contract Witfikan External E valuator ' 



1. 



Is there a legal reVi^ement that the evaluation 
be conducted by an outside agency? (If yed, j 
ini^ate the' contract; if no, go to Item 2. ) j 



r 



Are financial .resources available to support^ 

* * - -^-^ — rf- - 



a" contract for in" external evaluation?" 
yes, proceed to Item 3; if no, conduct the . 
evaluation internally and discontinue us^\ 
: of this checklist. ) / 




Ar.e credibility and objectivity of concern 
to the avfdiences for which the evaluation is 
conducted? (If yes, proceed, to Item 4; if 
p, discontinue. ) • ^ 



4< 




Is there an exterrlal contractor possessing 
•'tiie Beoessary, technical competenc^iwho is 
• available ancTwiiliftg "tbjio «ie ev.aluaftori.?*, 
(If yes, initiate the cont^act^4lap, conauct 
tE&^study. internally,, bu,t recognize the los 
. of credibility and objectivity.') 



/ 

/ 



Clffeck one foe each item 


Yes \ 


No ' ' 


\ 


K 


< 
• 


\ 

\ 


X 


■ A. 
/ / 








> 


• 





— - ^- It ■ 

SUMMARY TO SECTION TWO: . ' ■ 




1 


•RnsAfi nn questions 1-4 above, should this ' ' t 






evaluation be contracted with an external ^ 


r 




evaluatpr? fif yes, proc/e.?d *to sectio^i thjJee . 


V 

\ 


/ 


of the cheqklist; if no,' discontinue its use. 


* < \ 


/ 



"* Further detail on this criterion will be provided in a subsequent sectioii bf this- checklist. 
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D> Rationale for Specifying Contractual Procedure 
The utility of external , evaluation contracts may be more app^^nt a% ^ 
.this point than the utility of devoting time and space to a discussion 

; ^ ■ ■ • ■ ' • 

^specific contractual procedures. At least some adminis tractors and profes- 
sionai evaluators^ have commented privately to the authors that they see 

' i - .X 

little need for directions on how to write .evaluation contracts — after all 



to vrite one to cover an evaluation study. Perhaps, but things do not . 



they assert, anyone who can write a* contract in general should be able 

\ • ^4 



seejn quite that easy to many who. have tried to draft evaluation contracts 
and' later found them to^ be unsatisfactory and i^^^able of preventing 
the kinds of ^difficulties they were ijit ended- to forestall. In a recent 
'Activity, Stake (X974) compiled the responses of prominent ^valuators 
and researchers* to seven key questions about evaluation contracting. 
The responses of these, persons, made it clear that even ^perienced 
evaluat-ors and researchers see- the need ,for clearer contract statenrents 
.and specifications than has been typical to date. 

There are at least three major reasons why it seems useful to reflect 
at a mare general level on minimum essentials fdt^evaluation contracts .| 

First, neither evaluators nor clients of eyaluators Jiave accumulated 
much' experience Un drafting (or complyiCg with) evaluation contract^ 

■ beyond those. Situations where funding gyidelines automatically specified . 

* . ■ ^ X 

.^the nature pf the evaluation contract. In the latter case, the inadequacies 
< in these' guidelines ^t^ave* led rp inalrkedly in^deguate evaluation studies, 
making -it clear that- better, contract -specifications^ aiste necessary. In ' 



V 

the i^jority of cases ,* however , no guidelines have 'been imposed and* 



49 . 



46 



contracts haVe either been liion-e^ist^nt or*' inadequa<se. -Although the 
authors have perused a few very good evaluation contracts or "agreemenj:sO\ ^ 
of understanding"^ among the patsies involved; most see^ to be characte-rize 
by numerous omissions of importah^ points ,* lack of specificity or -embigUrit^ 
of specific terms of agreement, or "^irvternal conflicts among the contract * 
specifications. These conditions most li&l^ are attributable . to thet . 
fact that developing evaluation contracts ^s |:er^^^^6ognito for.iqgst^ 
evaluatOrs and-xlients . Efforts to be^ln to ou^ine at IWast the paj^a 



meters .evaluation contracts .should be w^l received by both K^ps. 

■ ■ ' V . ■ ' ' ■■ 

Second, ^valuators and clients often have very di^f^rent expect34:ioriejf, 
for the study, mal^hg it imperative that clarity be established, about ^ 
_ evaluative questions to^^e Tacldre^sed , procedures to be i^sed in. the-, study ,^ 
pTodu.cts will be produced and ox!x ^hat schedule, who has righcS to 
release the^^data', and so forth. Failure tb achieye clarity on.thfese aiSj^* 
\reiated matters . of ten leads to Ndis'appcfitrtment or disenchantment (or- woirse)'* ' 
oil the part of on^ or both partiejs. Most of the^ dt^putes^ surrounding,^' j'^ 
eval\iation sttidies arSt traceable toNfailure C^-^bek, su^f ici,ently precise ai^ 

\ \ ■ .A • ' ' > ' 

to achieve shared understandings and agreements that**guide the a<itivities 

\ \ ^ '>l^ " ^ V- . ^ 

. ^ ' ' \ \ \ ' ^ 

of the evaluator. in^c 



.conducting tjie study and the client 4n its 

^ ■ \ 

Brickell (1975') has\provideav excellent examples of difficulties in 

client-evaluator rel^tionships\hich could*have been 'avoided inmost 

instanced by clearcut contijactual agreement s . It is reasonable to hope 

that efforts to specify procedures ana\ standards for evaluation contracts, 

coupled. with feedback from those who attei^t to use th^m, would lead to 



r 



greater clarity" in thS^se :y^as .in ^the ;f uture. 



Third, there is a need to incr^se awareness on-the part of evaluatdrs 



\ 



and clients alikd of che necessity f or\specif ied procedures designed to 
provide necessary flexibility in evaluation contracts, ^.^e concept of" 
specifying procedures for flexibility may appear to be a nog sequitur , 
but on closer examination that appearance is» belied. - Evaluation contrac&a 
should carry within them specif:(/ed points at which decisions vcan be made 

by one or both of th^ parties about how the remainder rtf the evaluation 

• J' 



X 



will be^ conducted*. For example, it is stated later ±xi this paper* that 

^ r 7 . ,^ f ^, ^ 

* evaluation contracts should generally include either^^yCl) sonre provision > 

* for afi initial phase during which the contractor completes and pr^esents 

. ■ -I ■ ^ 

^the cli<ent a Retailed evaluation design which includes , all Conditions 



tacollection^ client responsibilities, timjelines, and' th^& like 




^^ir (2) provision f oV the/ f)otenti^^contractor to^ be paid >^^^e t^ prdpare 

^:*a detailed design to be used*^^s a basis for deciding final aw^ding'of 

the ^;^crn tract. In 'the /first cas^, imnte^iately following the submis^on of 

the design tliere should Se a review by th^-^client^,' followed .by a joint 

^^.ecision of contractor .and client as to whether to proceed with the 

/ ' , . ' ^ " 

remainder of the contract. 'In other words, the contract should contain 

/ ^ ' L ' 

Si, ' « 

* specific decision- points at which the contract can be terminated if n 
dissatisfa^tflon arises on the' part of either party, ju^t so 'payment fpr 
time spent a\id work completed to that point is assured. lk-*the second. 



the design is completed for the established fee prior to award^ing the 

nnai: jconti^act.^ -'-^Either accomplishes t!he same purpose; it; is se'hseless* 
to locV.l^th parties intp^ a contract which requires thei^ to .continue\fr 
one phas^ pa the next ^a^ ter* it has befcome clear that there is dissatisfaction 



on 



one or/ t^oth ^lart^s with the activj.ties or performance invplved . Yet 



\ 



1 



~£,ar too "of ten failure t9 think about* the need 



i for flexiMe phasing- in 



contrabts leads to 'preciseljr this" type" of 'unhapj)y ^.siCudtioh 



A second type" of' ftLgxib.ility ,of ten peglected ijv evaluation dontracb^ 



ip provision for mutual. alternations un the design,^ procedures or, reporting « 
of results.* D^spl^te the bes.t efforts of ^ contractor to include in the 
design'of the 'st.udy provision for all, possible ^contingencies , something 
will inevitably go awry.^ The ironic '^laws" .("if someJ^hing can 
'go wrong, it will" and "things talce longer than they do") seem always 



at least partially in effect • Unanticipated events and problems ^of ten 
force alterations in procedures if the evaluation data are to be useful. 
Contracts, must ^contain flexibility for mutual negotiation and decisions 
to acCOtmnodate such changes. Here is, one area where the competence of 
the evaluat^r^is most essentiaX^nd noticeable.) for it fi^kes a rej^ 
'•pro" to make the type of cre'atlve-adjustments ^whjLch are sometimes 
necessary to salvage an evaluation which is threatened by emergent 
problem? or events which make it impossibie^to^carry out the original 
design.^' This is only the first of several points at which^the qualif icatioi 
and genuine expertise of the evaluators (discussed ill the next section in 
greater detMl) become paramount to the s^uccess of the stidy. 

Qonsiderat ions such as the three abol/e' suggest, that discussion of 
standards £.nd procedur^ for developing and implementing evaluation 

L 



^This dols not mean tl)at^^ery evaluation can be Salvaged. Sometimes 
the evaiuatjioA can be so compromised by events that there is no chance 
whatsqever of farodoicing usable or\believable data. In'siTch instances it ^ 
is incumberit oriV^he evaluator to.^^ acknowledge this fact and terminate the 
evaluation !(which'^ain should\^^ po^ible within a well-written contract) 
rather thari to continue to expend the\lient;*s resources in a, hopeless 
activity. ^ ' * \* 
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contracts (i.e., the.cetAtent of this pap*^r> may Have^considerable litii/ty; * 

, - - ' ; .■ ^'v; .. • .r. ' . •/ 1-' \ . 

if only CO raiae the awareness of eval^Oators and iClierfts abaut impor |:ant,^ 
aspects of contracting which presently! seem- £9 escape* thre-'attention of ^ " 
^any who actsmpt^.to initiate evaluation contracts.. ^ • , 

, An additional pdint implicit in the above discussion should be made 
more explicity • Neither /eyaluators 'nor 'evaltiation clients are typically 
well trained^ in' contracting procedures. Further, there is no reason to 
expect th at eithe r party vill b;i$ well-yersed if^ the area of specialisation, 
of the qther; %t ds as unreaionable/to expect the evaJLuator ' to be knowledgeable 
about the intrictacies of school .finance as it is to, 'expeqf fhe schbol 

' '"\.' """f-^^'.T^ • .' - 

administrator to be' grained- ineev4j.ua6^iQjn, me thod,ology • Thef efoxe, the 

contract can^act as l^poir^t .of connnunication td facilitate mutual under- 

* ' 
standing and avoid either -pM-ty expecting (through .naivete) mbre than 

• ' 'J ^ "* 

is feasible of the of her* party, "'^'^^s, consideration of all a^^ects of > ^ 
contracting can *serve to protect both plp^ties and to improve the process 
of evaluation and, thVough it, the qGality of edttcajtional progr.ams. 
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E. Standards and Procedures fot Selecting Evaluation Cqntrac tors " 

- O 

ice an educational agency has decided to initiate an evaluation and , 
to contract* it with aa external evaluator, the itnost importatit activity 
Training" is to identify*,^nd ohtdin the services of an individual or agency 
with g^oine^ expertise in evaluation — someone with the %chnical competence 
necessaty isd do the job well-. As important; as contracts ar.e, even the 



^Por^ions 
which was i:onc 
certification 




draw heavily on an earlier paper (Worthen, 1972b') 
ems^ and potential of the notion of establishirig 



ional evaluators, 
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ideal contract will not compensate for the ^choice of an evaluatpr who 
is incompeteht to carry out the contract provisions. 

Unfortunately, marginal co;mpet:ence and downr^ighx incompeten(i^ are 
,far too common among many persons who purport to possess evaluation 
^pertise* , To defend this assertion, it may be helpful *to look briefly 
at the kind of persons who fall into the large group of ^extant "e^^^luators . 

'* ' • ' ■ . " ' " 

' Of all the persons .currently conciucting evaluations of educational 

^Jrogirams , 'only k small fraction have ever had any formal preparation in 

.evaluation methods and techniques. . Furlong (1970) found that only 30.4 

p^erceSt- of .a sample of some 300 .evaluators on ESEA Title I and Title III 



reading, programs had ever 'had any formal' CjOursework in evaluation.. He 
questloneci ^whethet much real evaluation of the programs was taking place. > 
Informal ^'observatiQU would lead the authors to' conclude that Furlong's ^ 
obseirvations are still applicable to a large proportion of the evaluations 

s , ' . ' j ^ ^ " 

conducted urider those" titles "ks well as liinder other auspices* 

Miny persons practicing a3 evaluators realize their academic prepar- t , 

*' » f ' * , 

'ation.is inadequate or irrelevant and do everything possible fco- increase 
their- evaluation knowledge and skills throuA in-service training, oppor- 

tunitie^ or on-the-job "boot-strapping" through self -study. Despite 

\ ^ . - , ^ 

^uch efforts, evaluations conducted by such, persons often suffer from 

\ ' . ' - I , „ ' 

inadequate conceptyalizatidti,' design, or an41ysis . . Many other persons 
;j(ho have go^d quantitative Skills (e.g., statisticians and pgychometricians) 
assume — gratuitously in the opinion of the authors — tiaat their training 
automatically provides them with the relevant expertise to function 
effectively as an evaluator. Such assumptions often result in evaluation 




studies which, have sophisticated and impeccabie^^-q^antitat 
but produce 'little or no useful information. Such persons ^f ten conduc 

evaluation studies which go unchallenged by practitioners because 

" ^ ' •* 

their , methodological '^complexity. Some of thesg quantitative -specialists 

are lulled into the erroneous but sincere belie*f that they are, producing 

good evaluation studies, whereas the advice they give or the Work they 

. • ** * > , . 

do is frequently useless (or worse, misleading). Unfortunately, there 

' ' ' ' ! 

are also those who know their advice is hkd and th^ir wort" inadequate 
but who, for a»- variety of^ motives, continue nonetheless to seirve as.* 
evaluators or evaluation consultants. 'The reference here is to the 
unscrupulous who commit atrocities in the name- of evaluation and depend 

on educatoi;s.l lack^of methodological" expettise-*t<^reiiciin undetected. 

* • . ' 1 ' , ' ' 

tThe point 'of the discussion so f-ar can be summarised as follows: 

a. Persons currently serving as evalua't'cif s range greatly in " . 

, ^ how well prepared they are to conduct evaluation studies, 

with?-^<teaaibrity being inadiequately prepared; 

b% Educators bften flnd*it dificicult to 'differentiate betweien 

^^ose*who are well prepared and^ those who are not; and 

c. The evaluators themselves cannot \>e depended upon to judge 

'how well qualified they are — some don't^khow and some 

won't tell. • . ' 

Against such a background it should be patentMy clear that some 

mechanism is' needed to provide administrators of educational programs 

or projects with reliable information about the evaluation competencies 



of prospective employ^ees of consultants. SucK iilformation would 'better 
enable" administrators to select persons who are adequately quali^p:e^ ^^^n ^ * 

luation work, * ^ ^ ^ ^ , • * 

The mechanism which , has been used in other* areas of education fo-r 
similar purposes^ is certification. Evaluatrn^-G^ri^fi^ or cr^dentlaling 

f^ some type has been" suggested by several evaluators .(e.g., Wor fihen7*"*4:9^Z2^ 
AshburTir^^f^TrS^Jie^ 1975) as a possible solutixDn to the problems pclsed 
above- The advantages some typ5"""^&^^a£54iication or credent ialing (or 





even "licensing'*) procedure would hdve for the consumer of 
services should he obvious. Certification of evaluators would allow 
educators to at leasQ, know who the certifying a&ancy viewed^as competent 
enough to receive^ a license to practice evaluation. The stamp o£ approval 
implicit in an evaluation certificate would be helpful to the administrator 
reaponsil^le for hiring someone to f ill\ a position in evaluation or 

enlisting the aid'' of evaluation consultaffts. Certification 'programs would 

* . ' ^ ' ' ' ' 

provide a function analogous to the .Consumers Guidance Commission with the 
iertificate serving much ^.ike their "non-hazardf)us" rating or the Good *^ 
HQ\^sekeeping Seal of Apprpval/ Administrators are obviously not prohibited 
from hiring uncertified persons. for evaluation jobs any more than consumers 
ark obligated to purchase products approved by Good Housekeeping * In 
either case, the information is there for the consumer's guidance and pratection 

Qertification* also holds advantages for the competent evaluator. At , 
present, educators are o'ften unable to differentiate the well-qualified 
evaluator from tHe unqualified. Similarly, it is difficult fbr educators 
to distinguish between two persons without formal ^preparation in evaluation, 
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even though jDne of tlvem may have engaged in bootstrapping to the* point where 
he or she is well qualified as an evaluator. Som*etimes even the rank 
iaposters are haxd to sort out from among the better qualified. In such 
a context , 'incentives are low for ihe more competent persons • Certifica- • 
Cion of evaiAiators should- bring with it the prestige and other rewards 
which typically go with a license to practice any profession where (a) high 
level competencies are required, and (b) membership in the pr9fession is 
dependent on pos,session or acquisition jo€ .those competences^. 

However, there are also' potential disadvantages, limitations and 

/ 

problems in certification systems which mu$t be overcome before evaluatoi^ 

^certification can be beneficial to the prof essdon. Resolution of these 

problems is a tiine-consuin4.i>g process that is likely to take at least\ 

ft 

several more years* before any significant effort to certify evaluators 
would be in'operation, and i,t fs entirel3r' possible that disinterest in or 
resistance to such a system may preclude its ever becoming viable in the 
field of education* J ' ^ ' * 

Whatever the long-range prognosis for evaluation certification., it 
is clear that at least in the interim, evaluatipn clients will have to * 
,'depend on other methods for determining thje competence of potential, 
contractors to conduct the evaluation stucfies. ' The remainder of this 
section contains an effort to outline standards and methods., which clients'- 

, ■ \-- 

might use to ma^e such selections. 

• I • ^ ■ . 

"* 1. Three Possible Approaches to Selecting Evaluation Contractors ( 

i ^ ' » 

Jhere would appear to be three major bases for making judgments about 

the qualifications of evaluators and, therefore, evaluation contractors** 

Each of these is discussed briefly below* 



* in 
!♦! Foxtaal training of> the evaluat;or» > Although far from a ' 

sure-fire criterion, there^is merit in' examining the formal training of 

^ . , ■ . . . i\ 

the ^valuator to determine if it is ^in' evaluation methodology per se or, 
if not, at 'least includes cpursajyork^^in areas in which the eyaluator 

I /. ' <A 11 

should possess kt least a Aodicmi of knowledge -ana skill. It* is always 
Dossible for persons to accumifl.ate many credit hours in a field of study 
wirtiouc becoming very knowledTgeable or facile wifih the content or methods 
of chat field. Too many peirsons go through too many university training 
programs <Jithout o^ng dei^on^trably changed 'to pla^:e complete faith in 

J / 

the meaningfulnes^of ev.jfen a graduate degree in a particular specialization. 



It is equally possible/for able persons to become competent evaluators 

without benefit of fatmai training in evaluation methodology, Y^t, in x 

the absence of mor,e/ valid indicators, it would seem appropriate to give 

considerable weigtft to an evaluator's formal preparation for the role. " 

The fact that parsons may learn skills on, the si^e is^poor justification 

for ignoring tbie ostensible efforts of some to pursue development of those 

skills by mpofe #f ormal means. ^In the^ absence of better me«ns, there- is 

considerab^ merit- in looking, at^the^f ield of study in which a^.'per son 

-holds ^the/degree, the credit hours accumulated in i^valuation, or the 

presence^ or absensje of specific courses or course sequences . ^ It a'lso 

seems wise to ask whether the training was taken with mentors who themselves 
c ' * - • . • ' , . 

possessed^ any particular credentials or reputations as evaluators.^ * 



ID ' ' * * • • 

This*and subsequent criteria stated as applicable to an individual , 

evaftuator should also be cans^trueci ^o apply to the personnel of evaldatiod 

agencies .under consideration as . potential' contractors , espec'ially persons 

who would play a major toxe in the evaluation. 

11 / ' ^ ^ ' • ^ 

•y •'••'•For' a listing of arrange of competeticies relevant to evaluation, the 

reader is referred to a/prev^ious worlc of , one of^the authors (Worthen, 1975). 



, 1.2 Evaluatio'n experience of» th» evaluator . Here again the 

" - ■ ^ ■ A- ^- ' 

criterion has limitations. There is an old saying in education that it^ 
is hard to tell whether -a^person who h^s been aSsteacher for ten years - 




has had ten years Of teachi.ng experience or one yeax- of teaching experience 
ten^ times. It is equally hard tt) know how much weight to give to one 'S 
experience as an evkluator or 'evaluation ponsulfant. Worse yet, some ^ 
experiences ^r^ decidedly detrimental in th^t^ they can' delude a person" 
±nco_believing he is dain^ evaluation when such isVot the case. Some-* ^ 
'^^people. have carried the title of evaliiator, in some tases for years, 
without reaiiy ^engaging in activities whicji qualify under the rubric of 



evaluation. Yet again, in the ab^e'nce of better meas/jrea of qualifications 
iris probably prudent to weight positively^* the fact/ that a ^potential- 
evaluatibn contractor has had experience in eva-luatiton. The probabilities, 
would ^eem heavily in favor of that contractor providing more help than 
one witH -nb prior evaluation experience (other conditions* being equal) .^^ 



/1. 3 Perfprmance (track record) of the evalt^ator . , This * criterion 
is sc^mewhat' more direct Chan -either af the two discus^d abov^e* How'well 
a person performed -as an evaluator in prior studies /Shou^kd be the 



''"ultinj^te oritejrion i^iL determihing whether or not he or she should be 

" . • - ■ / ■ ' r : 

granted Unqtljer contract to conduct in evalt^ktion. An obvious limitation 
.^of this criterion is that it might pVejudio/e the de^^siorT^gaiust^ 



newcomers in evaluation — persorls fresh frdm a doctoral pr^pgram in education 
seeking first jobs ^or a -new evaluation contract agency staffed by competent 
evaluators but wi^hput an established institu**Cional trac^, record • This 



12 - ' <o 

Obviously the criteria must bQ 'combined and trade-offs begin to 

emerge. It would seem pref erable^^for ^examplfe, to* contract with' a novicje 

''who t^as trained thoroughly in evaluation rather than a person who has no 

formal training in evaluation byt^has served for a year^ or"^ two in a 

minor evaluation capacity/ . • ' ' , ' > • ^ • 



limitatioh;is far oliDweighed by the advantages of this criterion, however, 
for its judicious application will probably provide more protection for 
the client than can ^be provided by almost any other step in choosing^a 
contractor, . , • ' ^ ♦ 

Perhaps the best performance measure in evaluation'would* be to examine 

; - ^ 

work samples in the form of wtltten^.evaluation report^ or other pfroducts ^ 

of .the previ^ous evaluation activities "^of the .evaluatoi: . This would be 

^especially useful if the previous evaluation wa^ in any way similar, to 

that desired' by the present client. Perusal .of such samples of previous 

work, should be most helpful in enabling the client, to predict whether ' ^ 

the evaluator would- be likely to conduct the evaluation* and report the 

results in a satisfactory manner. 

Some cautions are necessary in basing judgments on work products, 

fhowever. One problem which emerges immediately is that many evaluations 

are essentially group efforts and not the work, of one person. The us^ 

of such products .as a basis for decisions <abouto whether to contract 

with only, one of -*the individuals involved could be misleading in the 

absence of any reliable way- of determining the relative contributioti of 

the various authors. Even where the report is the result of one person'* s ^ 

efforts, this leaves unanswered the question of how to judge. the quality 

of the work if the technical aspects of the study exceed the .'technical 

knowledge o'f 'the potential clilent . Reviews by other expert ^valuators 

would serve, but the feasibility of such an g^pproach may ''be questiorlable 

for most clients. There '-are also instances where competent ev&luators ^ 
* . » . . 4 « 

produce poor reports through no fault of their own but because program 



r 



*personnel with (or for) whom they were working would not cooperate in 
♦ carrying o'ut the evaluation'design. These piroblems do not negate the 
use^ of work samples'as a criterion, but oniy argue that it must be applied 
sensibly and may not always be sufficient in and of itself 

^ ' V Z \ 

In many situations it is not possa^ble to obtain samples of previous * 
evaluation work completed hy potential contractors. Few evaluation studies 

. V. ' 

find outlets in professional journals and most are fugitive docume^tsf. 

unless syppli^d by the author^. Even that is not always possible becau^ 

of .resti;ictions -on distribution of results of the evaluation (possibly "'^^.^ 

a restriction imposed by a- contract under which the previous evaluatior^ 

was ^conducted) . In the absence of ,work samples, references from" former 

clients would l^e most important. Indeed, they should probably be sought y 

even when work samples are available, fpr relevant information might b© 

obtain^ed which is unavailable in the products themselves. Obviously even^ 

this -criterion is no protection against the unscrupulous evaluator who 

/\%ss:s as a reference a supe;rintendent *f schools for which he has conducted 

previous* evaluations without making mention 6f the fact that the super- 

intendent 'is his father-in-law. The earlier discussion of the need for 

standards of ethical practice for evaluato.rs would be relevant here. 
\ . 

It should be noted that references vfrom clients* are not restrict;4d 

■ ' . ^ I '-- 

to personnel in "the progranf which was evaluated. The funding ag-enc^ 

*. • ^ - . . ^ / ' 

which initiated and supported the study or secondary audiences (e^g.*^, school 

boards) are equally appropriate sour^ces of client judgments -about the 

quality of a contractor's work in evaluation. ' 
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iK^eicher or both of th^ above performance measures are used" in 
selecting'an evaluation contractor, they should go far toward avoiding 
serious selecti^^ errors • Use of such criteria is also important in 



making a selectiojii^Jj 
individuals and agenc 



jn proppsals from potential contractors^. Numerous 
ies seem to have'^Far^reat^r 'talent in producing 



convincing proposals than they do in delivering oif what'they promised. 

/ , . . ■ ■ 

The use of ■ references or worlc samples Should Kelp to identify serious 
.gaps between wh^ a contractor purports to be able to do and is able to 
do in "fact. 

\ 



\ 



T 



< 






CHECKLIST: SectionThree: Criteria to Consider When 
' Selectingyan E valuator 



(Each item below is written to apply to an individuaL If the potential contractor is an ^ 
agency the question should be re-cast accord ingly>). * ^ . " 



• * 

1. To what extent does the formal training of - • 
^ the potential contractor qualify himAer to 
Tcondudt evaluation' studies? . (Consider 
* ' tten?s suqli as major oi* minor degree 
' ' ' speciaUzatijon; specific courses or course 
\ . \ * t^., ^.'^ei^uences^Ln eyaluation methodology; 
• '"'V/ - / * 7 qualifibi'tioiis of perSons with whom 
^ ' \ y • 'tSraining^ytas taken, ff such qualificatidns 
V *' o ' ^ * ' are Ipiown. > 


Contractor appears to be: 
(Check one for each item) 


Well 
Qualified 


Cannot 
Determine 
Qualifications 


Not 
, Well 
/qualified 


a 




m 


s '^V/.. ' ' qualify him /lier' to conduct evkluat ion 

""^^Z \' ^sfeidies? ^(Consider items such as leiigli . " - 
^ ^of ''exnerience;' relevance of experience* ) ^ ' 






V 


. . . ' ' \ ^'2.^' * To whiat exteiit does 'the previous performance i 
. ^*;^ . ^ ' ^.of the [.potential evaluation contractor qualify | 
^ * . . ,;.'him/her to/3onduct evaluation studies? \ 
' - ^ - - > (As judged by work samples or referfenceis). 

.:• __ i '. — v_ 
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SUMMARY TO SECTION THREE : 



Based on questions^ 1-3 above,, to what extent /is 
the potential evaluation contractor qualified tio 



conduct evaluation studies ? If "well qualif^fed, " 



proceed to subsequent Sections M this checklist; 
If *'not well qualified, " discontinue consideration 
of this contractor. If "cannot determine qualifi- 
cations" on basis of initial information,, seek 
additional information arid/or assistance from - 
colleagues with expertise in evaluation to make 
this critical judgment. ^ . 




NOTE: A neutral category has been included in thi$ section of the checklist becai^se informatlot 
initially available alJout potential contractors may be insufficient to petTnit unequivocal • 
judgments to be made on thiB dim ensioL . - • » 
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2. Procedural Options, for Letting an Evaluation Contract * 

The preceding criteria can be applied within one^ 
alternative procedural frameworks for letting an evaluation contract T 

In the firstj the criteria are applied to One or more potential. 

contractors and a si'ngle contractor is then selected as the person or 

■ ■ . ' r-. , \, ■ ^ . 

agency with whom contract negotiations are initiated. .Negotiations 

with* the» single^ individual <?r corporation continue until a mutually^ 

satisf actary arrang^ent ig? reached or negotiations are broken off and 

another contractor is selected (again, use of criteria would be > 

appropriate). This approach is kno.wn as sole source- contrac^ting . 

V * - In fhe second approach, one solicits proposals from multiple parties 
^ . \ ^ • ^ ' ^ . ' 

and evaluates the proposals against criteria to identify the one best 

able to fulfill the contract. Here thg criteria from section three of 

jihecklist^ would b€ aj>pl-4i^d aTong with criteria for judging the^ 

specific ^BSTlity ~o?'~rtJe*.er^ When' one •contra.ctor has been selected 

on the basis of which proposal ano^aoiitracfo^r qualifications best meet 

the prespecif tgd criteria^ serious negbtiabfcQns are then Initiated . This 

approach is known as the-T.eques,t for proposals' (o^v^^) approach.' 

The third approach' is'^'^b^cen i^i^ most feasible optibqfor local""-'^ . 

school districts. In this approach prol&pective evalu^itbrs svbmit a 

brief statement of how they would proceed grVen the particular evaluation 

situation, in addition^ each supplies his or heX credentials and 'qti^lif i- 

catibns. Applying the section three checklist cirifeUfia and examining 

the proposed style o'f evaluation, the 9lient selects a Articular party 



' }^There axe variations of these thre6 approaches, but the authors 
deal here with phe essential characteristics. 
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with whom to negotiate. This approach has. been descr4'bed as a synopsis , • 
review approach 

2.1 Sole sour'ce contrac^ng . Sple -source contracting, is by and 
large the simplest technique and least costly in terms of time and resources 
required of the funding agency in initiating the contract. All one need 
do is to identify the evaXuator , using the criteria- in the previous section, 
and negotiate. It- is particularly ' appropriate^/h^re there is clearly 
one contractor who is known to be especially capable and responsible, who 
Jtiil do the work at a price, well within the* budget available, and who can 
coim)lete thk work by the* time the information is needed. In such cases, 
theffe is seldom any reason to look further, unless there is a legal 
restriction which mandates the use of an alternative approach^ However, 
if no such paragon emer&e^, there are several , means ^one can emj^loy to 
identify the"^ pool of potential evaluatdrs to which the criteria would be 

applied. An individual or corporation who has performed otheV fevalu'ations 

<» * 

of a high-quality for your agency is a prime candidate. So also i$ the 
evalUator who has done similar work elsewhere and Xeceived approbation 



IT clieikis, i^., w 



from his or heij clients, i.\., word-of-moutb advertising. A third, approach 
would be to use a' panel of advisors whose job it would -be to recommend 
a particular evaluator . * ' / * ^ 

The discussion of negotiatiCn' issu4^s and procedures is deferred 
untijp^j^e next major section of the paper. 

2.2 The "Request for Proposals (RFP) approach . The.RBP approach 
is considerably more involved, but oft^ jystifies the additional expenditur 



^^The authors, are indebted to Mark Greene f'br suggesting the inclusion 
of this approach.; ' * ^ , * » 



and effort. Th^ ingredients of the RFP are somewhat standard. One 
must include a description of the context in which the evaluation is 
to occur. Is it an isolated rural school? Is it a federally funded 
profject in operation at several 'remote schools operated by different 
agencies? Is there a high projj^ortion of non-English speaking ♦students? 
The answers to these and other questions might well affect the evaluatorjs 
perception df hi^s or her ability to copduct ap adequate evaluation in 
that context and thus thedr inter^est in preparing a proposal for such 

a study* So as to- elicit responses from those who would be genuinely 

' * ' ^ ^ , 

inter^ested in working in the^particular setting and discourage the 

merely curious., the RFP ought, to provide a compreheiisive picture of the 

locale and system in which the .evaluation is to occur. 

The purposes and characteristics of the progrsam or *ptoduct to be 

evaluated should also be provided>\A particular evaluation agency may 

^mpl^t^d the development of a>s^ of elementar^ scho^ol reading 

emphasi2*e -phonic skills • They are much more likely ^-to 

^hich stipulates improved skills in this area than they^ . 

bcuses on general reading achievement. Clients 

ators what the project is intended to * 
to be implement-ed^ 

/^\ 

proposal the evaluatpr Is entitled to know what 
specific outcome^ ar;e expected from the contract. If th6 client, wants 
quarterly reports, that constitutes more work than an^annua^ relpor t and it 
will (and 'should) cpst mpre* If, in addition to ^the evaluation of the partic 
ular project, the client ^expects the contractor to work with local staff 




so as to \improve their , evaluation skills, the potential contactor has 

a right to know thag ^o all expectations are clear before time and energy 

are invested in'the preparation of ' a proposal. 

The pptential contractor needs to know what resources are available 

for the task. If the 'client cannot or will not spend more than $3,000 on 

<(i , * «^ 

evaluation, that inf oritta^ion must :,be provided. Otherwise, the bidder is 
r ' ' ^ 

liable to waste time and mon^y; in' the preparation of a proposal, that 
X \calls for a $15,-000 evaluation^ Indeed, if the potential evaluator i/s 
led\to believe that the stSdy is to bc^e a lar.ge sgale inquiry, he 5=or she 
c^uid^^i^est the equivalent of tTae '$3,000 in the preparation of the 
proposaLl ^\^s another instance, if adequate computer facilities and 
' software are ''available without cost to the' contractor^ that may affect 
.someone's decision 'to prepare a proposal. 

Similarly, if constraints exist, the RFP should be candid. If 
certain data will not be available to tho^evaluator , if the evaluation • 
must be wholly completed in five rionths, if the evaluat;or must work with 

a specific subcontractor, he or she is entitled to icnow that before.- 

rt » " ' ' ' ' , 

preparing a proposal, • . 

— ^ . 

In suimnary, the, RFP mus.t include: > ^ ^ 

1. An adequate descriptiion of the context in which the 
evaluation J.s« to take 'place 

2. Clear statements of the pur^ioses and characteristics of 
of the project to be evaluated 

3. The specific outcomes expect^ed from the evaluation - ' 

4. The resources available for evaluation 

^ r 

5. The constraints which would affect the evaluation. 




'*"*«^ 



\ Qrfe can choose to provide more inforination in the RfP ;*'■^ca•6iaaa^j^^ 
^it may make sensa_to restrict the methodology to be employed by the % 
evaluator. If, for example, the expansion of a project into a new 

geographical region is to be evaluated,, it woulci' be prudent to. evaluate 

\. *^ ' 

it in a manneV i^imilar to the evaluation of its predecessor, assuming ' 
thd latfcer was w^l accomplsihed . Indeed, one of the 'great faults of 

^ \ " ■ ' ' • ^. 

curitent evaluation practice is thatjso little of the knowledge is * ^^'^ 
cumulaC^ive^ ' \ ^' , . , 

In planning a particular project, expert' experimental design * , 
assistance might have been acqur1:;ed, an'd agreements for the installation 

• ^ . \ X ' • • ' . : 

of the project settled on. In such.^ case, the flexibility of the c * 
evaliiStor is 'limited and the RFP should make this fact clear. 
. V Perhaps political factors mitigate against tSie use of a particular 
^ kind of instrumentation^ Again"*", ♦the potential evaluattor is entitled 
to this knowledge pr>i9r^ to preparation o^f the proposal. 

In general, however, cautiO;n. shouldj be exercised in prt-specifying 
the 'methodology of the evaluation. There are two primary reasons for 
this 'recoiamendation. By requiring the' ^valuator to .stipulate the 
methodology he or she deems .appr.opriat|e, the cotitracting agency has a 
greater variety of 'alternatives "to select frpm. One of these alterna-. 

\' • * V / ' , ' ' 

. tives may^be both creative and uniquely responsive and thus, although 
unanticipated^ the chosen option. * Secondly, providing , too much methodo- 

logical, information 'invited proposals from those who have exc^ellent 

• ' ' ' * ^ ^ 

verbal skills' but little' methodological understanding'. This kind of 
contractor, writing a reflection of the RFP In the proposal response, 



may win the contract and be unable to perform the task. It/ is feasier 

to. identify the individual who lacks methodological sophistication, who 

knows "only of pre--post evaluations with all of the contamination of 

information tha't these often entail^ when the potential contractor is 

•required to present his or her own plans. ' 

Pr^ocedia^es fov' evaluating proposals.^ Jud^ging the proposals otie* 

receiv'es in response to the RF? can be made easier if certain procedures 
• ^. ^ . * » 

are^ employed. - Thermos t* important consideration is the^ pre-speci.f ication 
of the criteria to-be used in judging the/merits of 'the' propiosals.- 

Assume 'that the total ^pumber of ppints a proposal can receive is 

"""^^ * ♦ 

""-.^ * > * 

100. It is possible to illustrate how criteria might be weighted 
differently for deferent projects. Project A is a reading ^program. fo|^ 
children Whas^ native languagg. is not English. The children live l^n 
isolated rural .area^s . The teaching in the schools is biiingualt . The 
projBCt is federally funded with a fixed financial aMooation for 
evaluation. -The 'project requires rather sophisticated evaluation / 
methodology. Further, the funding agency has placed restrictions on the 
instruments which can h.e employed. Only those data collection instruments 
developed vunder a separatfe^contract issue4 py the funding agency to a 
^jor test developer are permissible*^ * >Jb! ' 

Project B is a vocatfbnal training program for iunior-Jiigh school 
' Students, in the EngIi«Shv^peaking. secftor of an urban s^ihool system.' The* 
^roj-elit. requi,te^ rather roiitine tli^thodological skills with the exception 
*bf somewhat higher l^yel Competencies required in the ar,ea o'^f sampling '*l 
p#theo.ry. .A prime consideration is the ability^ of the contractor to develop 



- applied^ performance tests" which can be used to evaluate the success of 

^*^the project. Moreover, the project is not very well funded and any 

savings in the evaluation sphere could be used to expand the services 

provided to .students , * ^ * ' • 

In Table 1, -illustrative relative weightings of various criteria 

are ''iuxtaposed for these two hypothetical' projects. The variety of 
' f' 

criteria 'which coul4 be employed, and of weights that could be applied 

i «■ 

is virtually limitless. '.What is imperative is that qriteria be * 
idetitifie$i prior to th,e competi'tlon and. shared 'with all potential 
contractors'^in the RPP.- • 

Of course, criteria by themselves are not sufficient. One needs 

•a means of applying the criteria to the proposals. Placing that decision 

*■ ( * 

in the bands of a single 'in'dlvid'ual imposes too severe a burden,. It 

; ' ^ . ' • 

is unreasonable, to expect any .one person to have 'acquired expertise in 

u * • 

all\ the relevant ar^s. A panel of reviewers screening* proposals and • ^ 

• •* 
a^iriving at a panel. juSgment is a preferable option, ^ ' -\ 

In ,the selection of a corporation or individual with whom to 

negotijate a contract the object is to choose the moS-t qualified agent 

" for 'the particular purpoises. That decision ^should rest solely on the . 
perceived* capacity of the agent to»perfonn the requisite tasks. For 
that reasd'n,s .panel membership should -consist only of individuals m 
(internal or consultant) with expertise in a "relevant area. Later in 

-.<his p^aper we will discuss the need for the involvement df a wider, 
spectrum oi individuals in the negotiatidn of the contract. • . 
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TABLE 1 

Illustrative criteria weights 
for judging propo,sals 

\ . . 



X* . ^ . Proiect f 

Criteria • • ' • * « 

* « • 

Understanding of the problem ^ l6 ,5 



Merit of the proposed evaluation methodology "^35 20 



Instrupient de^?^lQpment exparierice ■ ' " — * 20 



Appropriateness of proposed timelines - 10 ^ - 10 



Clarity of the pijoposaj. 10 15 



Rural schools evaluation experience 10 



Bilingual evaluation experience ^ 20 



\Cost considerations , 5 30 



EMC ' : - ' 



♦ Aduanja^es of the RFP tcpvToaah* There ^re twa primary ^advantages , 

to the RFF'^Sppr^ach. ^Firsct it^ f)rovides the contracting par-ty a wider 

range of ''possible crptions to, choose from^in terms of", jthe evaluatian 

IS 

•Strategy to be employed. Given more alternatives, one should be able 
to reach a better iofonried decision. The sefcond major benefit is the- 
enhanced credibility of the evaluation that will,xesult. By minimizing 
the threat of "cronyjLsm" in the awarding of evaluation contracts, one 
increases the confidence others are likely to have inr^the reported result 

Obviously, the threat of a charge of racial or sexual bias is also . 

i" * 

St 

substantially redUced. ' \ * ' * 

Disadvantages of the RFPiapproaah. There are disadvantages to the 

'RFP route, as well'. It is more costly for both parties. The 

client must invest timeiand money in (a) preparing and distributing the 

RF? to possible bidders and (b) judging the proposals received. The - 

contractor must bear the costs of propos^al preparation. Since these 

costs can be substantial, one may be restrictlB^^otential oontrarctors ' 

to those agents which have sufficiently large capital resoiirces to penpit 
/ \. ^ ^ - ^ 

them to speculate in the proposal preparation markets- • Th^e is no 

• evidence to suggest that capitaT^^reserves are related to quality of 

evaluationi Indeed; for many evaluations, ^ the small scale independent 

* * 
agent may provide better and- **more responsive'** service. 



IS' . ^ 

It is ger^ral'ly Considered unethical to steal an idea from a 

rejected proposal and suggest that, the selected contxactoi; employ it.' 

In certain instances „^ the practice is illegal %s well^as unethical. 

If a client believes *that two of the potential contractors acting 

collaboratively would be the best option, it is legitimate to explore 

that* possibility wi'th them. ^ 
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A word caution is in order ac chis pSl!Itfr?*.JEii^ proposal is not 
equivalent j:o the evaluation design. Jt'is rather aa outline, a sketch, 
of what the..evaluator would undertake. It provides a basis for^ negotiation. 
The preparation of an evaluation deVs^n for all but the most trivial 
orbblems is a complex endeavor that ire^^res*a high level of training, 
exp^ffience and skill* The prof essio'hal evalua^tor desefves juist compensa-- 
tion fc^r his or. her efforts^ There are /'evaluators" who vill provide this 
service, free of .charge^ to potential clients. Those who consider cost 
alone are this individual's , legitimate prey* 

» 2.3* Synopsis review approach . Essentially this approach is a 
compromisa..between .the other two. One contafts potential evaluators by 
letter, brief ly -describing the evaluation need which exists. A brief 

. ^ ^ 

statement, perhaps 1,000 to 1, 500 .worcis , is than submitted by interested 
evaluators outlining how they would conduct the evaluation, alongNwith 



if Ic 



a statement-^of th^ir qlialif ^.cations . One judgfes the merits of these 

submissions and the qualifications 6f the individuals* Once a (Choice 

is made^ negotiations p^[oceed* . " \ ' ' 

. ^ ' * \ 

"ihis approach is les.s costly for both parties^than the RFP approach . 
and therefore more feasible in many instances. ^Similar to the I^P 
approach, it ser.ves ^to enhance the credibilitj^ of the evaluation and 
reduce charges of bias. It does not, however, permit the evaluator 
to devjelop ideas in suf :^icient detail to ^convey the innwative atragegies 
that might be used. Yet to require more would be 4:0 necessitate 
^greater . Ixpenditure of \the client's resources to review longer , 'more 
complex plans-. It ntlght,al$o draw unfairly on the evaluator's resources 



since if the funds are insuff f cienp to supp'ort an RFP approach, the, 
dollar amount of the evaluation contract is probably coo small to warrant 
a heavy investment pf the evaluator ' s ^i-iiie and other resources in an 
effoi^c CO win the cQncract* 

A caucion i^' alsp-^ order. There are those who have mascered 
the cerjainology arid semancics of evaluacion wichouc acquiring any methodo- 

■•' / ■ 

logical expercise* A brier stacemenc prepared by such an individual 
can, be very persuarsive in ' che absence of any . requirement » thac che 
evaluacor demonscrace knowledge and expfircise on che relevanc mechodology. 
The rhecoric of a synapsis is obviously no subscicuce for a wi?ll conducced 
inquiry > * . - 

When CO Use the Various Approaches ^ \ 

Given che po't^cial losses and benefits associaced wich eac If' abroach 
how does one decide whiqh Co use? One chii)g is cercain — che RFP approach 
can only be used when che' compecicion is cruly opeo» If che concracc 
is "wired," i.e., che mosc probable winner has been idenc/fied prior Co 
che receipc' of proposal?, or che RFP has been wricceti Co favor a, known 
concraccar, disascer lurks • In the firsc place, che number of proposals 
one can expecc Co receive in response Co fucure RFPls will dr:jop 
subscancially, for concraccors in any field are a 'canny lot who soon 
learn which agencies prepend co usq open RFP compecicion Co sacisfy legal 
requi-remencs, b,u{; really have'* che w'intier picked in advance all along • 
Secondly, thos4 who 'discoyer chat chey have wasced resources in rigged 
compecicion. are likely co make every efforc co.discredic che wotk chac 
does go ort» Finally,^' if ic can be proved chac che cpncracc was "wired», " 
legal accian mighc be inscicuce^., 



As indicated earlier, if a client is •curr^ntljj working yith 

/ ' ' ^ . \ \ 

evaluacor on a given project 'and is satisfied with the performance hp 

' ' ^ A" > • \ ^ 

date, a -sole Source* contract would seem appropriate • . Lt is also rea^nable 

' • '■ • * ' \ • ' •• 

to explore a sqI^- source contract with an evaluator who has cfr ^ >providing 
satisfactory service on another co'ntract, especially i,f costs can be \ 
reduced thereby, .e.g . , bySp^ouct ion .in travel costs viien the same^ ^ 
.person-trip can be used for mult^iple purposes. 

'\ One muse, however, be cognizant of the dangers of "cronyism." If 
the jf^requency of contracts between a client and evalifator becomes ^ 
e:tcessive> a loss of credibility may result. "^Outsiders may start t 
assume that tBn two parties have a "cozy relationship" wherein th 



evaluator is co-opted by 'the client's interests; l^iiis is not^td/argue 



aore common, especially for fed^erally funded endeavors. The presumed 



that external credibility should be .the only* dt even, the major conc'ern, 
but it is something' that one should consider. * / 

^ Legal restraints-agaitist sq^e source contracting are becoming 
^ally for f ed^^al] 

intent is to reduce the bias in contracting procedures whether that bi^s 
is deliberate or unwitting. Where this cons^aint exists, a synopsis 
revi^ew or RFP approach should be- adopted, 

^ The financial resources available ^ould^.be considered wh^n one is ' 
reaching a' decision on which option to choose. One can ^compute the 
hosts associated witli an RFP proc-ess — staff salary and b.enefits,- consultant 
feeSy production a^id mailing' costs, etc. Comparing these coats to the' * 
dollars one is willing or able to commit to the evaluation perfii^s one ' • 
to make a better judgment about Whether t;he benefits of the RFP mechanism 




are worth the costs. Assume /th^ the minimum expenditure for an RTP ' 
prepararion and proposal reyie^-is $500. It makes little sense to spend 
^hat amount of faoney if the total evaluation contract-will be $3,000 
or less. Maybe it. is only worthwhile if the contract value exceeds 
S5,000. There" is no single universal standard in either absolute or 
proportional terms • Perhaps the best acivice one can offer is the old 
^ adag^**you get what y6u paty for." The client should decide what cari be 
realistically afforcyed. Sole source contracting costs less. to initiate. 
*If^ this approach is /inappropriate,^ the synopsis review approach .is the 
more economical o|>tion' remaining for initiating a contract.' Although 
initially expensive, use of the RFP approach (if it can be afforded) 
may lead to lo/ig~range savings by identifying qualified evaluators 



with creati^^^ inexpensive approaches to condyftrting the evaluation. 
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CHECKLIST; Section Four: Cri^ria for Selecting Among 
Procedural Options for Letting Contracts 



* 

I. Is there a legal constraint against sole source , * 
contracting? (If yes, discontinue use of this 
section of the checklist and go to Item 4 below; 
if no. go on to Item 2.) 


Check one for each item 


Yes 


No 






2* Is there an evaluation contractor who meets . . ^ 
the criteria in checldist three, who is 
interested in this evaluation, and whose 
services are desii^ed by &e^^Siient? (If ye^, 
go to Iteni 3; if no* go tcj/item '4. ) 






3, Have jsuch a large nunf^^r of contracts' been , . : 
: let to this same contractor that "crony|sm" \ ^ , 
' niay be a cj?mger? (If j'es-, reconsider Item 2 ' v • 
above for a new contractor;'if no begin 
negbtila^tibns. ) . ^ . * - « 






4, Ara the financial re^ocrfces availa|3le for ^ 
this evaluation sufficient to- warrant the cost 
ofanRFP? (If yes, prepare an RFP; if no ^- , • * 
use synopsis review approach to find a 
- suitable contractor. ) 




■ 1 

-\ 



SUMMARY TO SECTION FOUR: 



Based on questions ,1-4 above, which of the 
following procedural options do you select 
for letting the evalilktion contract ? (Brief 



definitions of options appear below on this 
page.) 

(If "a'' or t"c" is chosen, go to SBction six of 
this checklist; if *'b^' is chosen, go to section 
five of this checklist.) 



Q So^ie Source Contract 
b. Q Request for Proposals 
^ 0. Q Synopsis Review Approach 



Sola Source = contractor identified solely on the basis of known qualification to conduct 
evaluation studies 

Request for PropossLls = contractor identified qtf the basis of a detailed proposal for the 
conduct of the evaluation, submitted in an open competition for the contract. 

«, • < 

Synopsis Review Approach = contractor identified.on the basis of a brief outline of (li^how 
' the evaluation study would be conducted and (2) evaiuator's qualifications 
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CHECKLIST: Section Five; Criteria to Consider When 
■ ■ Using an RFP 



1. Does the RFP include: ' 

a. an adequate description of the context in 
which the evaluation Is to take place? ^ 



Check one for each item 



Yes 



No 



b. clear statements of the purposes and 
• chaxacteristlQS of the project to be 
evaluated ? 



c. the specific products expected from 



the evaluation?. 



d. the resources available for the 
evaluation ? 



e* the constraints which would affect the 
evaluation ? 



If methodological restrictions (e. g. , 
reslciciigns on sampling) are essential,"^ 
i^fetheyincluded in the R FP ? 



3* Has a panel been selected ^or reviewing 
the pr^oposals receive'd ? ' . 



4. Has the panel established criteria for 
judging the proposals ? ' , ^ 



5. Are these criteria included in the RFP? 



SUMMARY TO SECTION FIVE: • \ 




'•5'- 


■ \ 

Based on questions 1-5 above, is the RFP 
':^eady to be issued? (If yetf, issue RFP and 


proceed to section six of the checklist; if\ 
tio. re-write RFP). 



ERIC 
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. F« Standards and Procedures fot Negotlatln; 
^ - With an Evaluation Contractor 

Once *an evaluation extractor has been identified, the negotiation 
for a contract begin. If the RFP approach was used, the proposal^ 
constitutes^ the basis for negotiant ions, but much remains to be decided. 
If the contract is a sole source contract, eve.n more may be^ required, 
This^ction of the paper deals with six topics:*^ when to initiate 
negotiations, the evaluation design phas^^of negotiations, the scope 
of work, itesues for mutual discussion and agreement, who is involved 
in negotiations ,• and when to abort negotiations • 

I« When to Initi^frekTiS^egotlations - " • 

It is by now a truism that if evaluation is to have maximxim impact 
it must begin early in the life of a projects It should then be self- 
evident that the negotiations for such evaluation must occur as soon as* 
it is practical to do so, i.e., as. soon as the commitments necessary 
to pr^oceed with the. project have been made. ' % 

GiVen the nature of summative evaluation, some have concluded' that 
it is -appropriate to engage an external ^valuator only when the project 
has been up and running for a considerable time. The evaluator is tt^en 



expected to come in and collect impact data f rom/teachers, students or 

• \ • > ' \ . ' 

whomever an4, prepare a final report. At first glance this approach 
may seem almost*- logical . . In ^int 'of fa^t, it creates problems, of the 
greatest magnitude." ^ 

'' "T«here are^ basically ^two dimension^ of the summaitiVe evaluation.'' ^ 

. ^ y ' ■ . ' ' ' "^V *' • * . '* ' 

ji^tjich determine whan the' evaluator- should ,be, engaged%^ . The first bf 



these is the type of inquiry involved. Is an Sperimental desig1?i, or ^- 

^ ' \ 

one of the fime frame approaches (anthropological study-, time §erl\es, 
longitudj^nal case studies , 'etc • ) most appropriate? The second critical 
dimension conc^erns ' the responsibility of the eyaluator for the creation 
of new instrume\^ts or data collection procedures. 

In the instance when an experimental design is to be used, the 

evaluator must be involved prior to the assignment of the treatment to 

4 , 

the experimental units . The complete justification of this statement 

\ 

\ ^ ^ 

would require an examination of experimental design which is well beyond 

the scope bi this papei;.^^ Suffice to state that any outcome is the 

consequence of the interaction of treatment and subject. The corroaive 

effects of acids depends* to a certain extent on the substance with which 

they come in contact. In much the same way, the impact of alternative '^l 

approaches to reading, for example, will depend upon the status of the* 

children taught. Unless the evaluator has an opportunity to influence 

/ * * ^ ' y » 

which students r.eceiv^ what instruction, he or she will never be able to > 

disentangle those effects which' arise from treatment differences from - 

. \t 

those due to the' characteristics of the stXidents. 

* Contrary to ^the usual expectation^ the use of a time frame evaluation 
approach requires an even earlier^ ?ntry point. In these cases the 
evaluator must be engaged in time to permit' the collection orf data 
concerning the status of those who are to participate well in advance of 
the introductipn of the treatment . In an anthropological cas^ study 
approach, for example, the evaluator must. be. on site and recording the \ 



•''^The interested readey is referred to* Campbell and Stanley (1963). 



nature of the phenomena of primary interest well prior to the implementation 
of the innovation so that he or.^he can be^in a ;positioTV, to note the 

¥ .. .. ' ' 

changes which occur as^or^i^-ter that event t:3^kes place. While there is 

<^ 

, no set standard for when observation begins, six months to a y^ear prior 

to initiation of the treatment is not unreasonable for most projects; 

' ^ , <i \ 

The need for the development of new instruments imposes^ an additional 

. 

lead time requirement* In these cases the evaluator should be. engaged 
soon ei\pugh to permit the development, tryoufc, analysis and revision 
of the instruments prior to their first intended use * ^There are few 
things less satisfying tl^n reporting the results of a weir conceptualized 
^evaluation which is inconclustve because insufficient time was allocated 
for .the preparation of valid and reliable instruments. One circumstance 
which qualities as less satisfying is reading that inconclusive report 
when you are the client for whom it was prepared. o 

2. The Design Phase — * . ) . * 

. The importance of the design of the evaluation' has already been 
explored.' It is ' indeed the most critical element 'in the^evaluatOr * s task 
because all else depends on its logical consistency, comprehensiveness 
and clarity. During the period of negotiations, it is recommended that 
the evaluator submit a detailed design which can be reviewed by the 
clients and modified as circumstances and mutual satisfaction di^ctate. 
The resultant' design would then be part of the contractual arrangement: 
agreed* to by both parties.^ . - <r 

As stated earlier, the evaluator is, entitled to compensation for 

o 

this ^effort. A fee should_be_^stablished for the preparation of a 



8i- 



detailed 4esign and .this fee should not be contingent upon whether or not 

the design is accepted. The selected evaluator, f of example/ agrees 

foi? a fee to prepare, a design for the evaluation of an innovative^rogram 

in career ^ducation^which involves the assignment of students /to various 
• . & ' ' - * 

o . • ^ ' , 

employers for a portion of the school-day* The- evaluator insists tl\at 
thfe selection of students for participation in the program must be rand6in 
within^W--v<>Xunteer group in order for the results to-be credibly*. The 
client, for valid political reasons j o^'cannot agree to that condition,* 
After discussipn, a mutual decision is reached that the client should 



« ^ 



seelc a new evaluator-. Even, though the first evaluator 's design w^s 

; ; ' ■ ^ 

rejected, compensation for'"ti|ae spent on its .preparation is in brder/ 

, * ' By adhering to this practice, the client is ultimately benefitted. 

With the knowledge thart*the fee is secure, the evaluator will design the 
>^ ' a - ' . . ^ 

st'udy that, in |iis. or her prof es.sional judgment, is best for the project 
within, the xesourte limitation. There is no motive for trying to r.edluce - 
the risk of financial loss. by minimizing^ the investment in the design, 
which is usually accomplished by trying to figure out "what the client . 
will buy" father than what should be done: In shiprt,> the. design fefe- is ' . 
one's best pLi:otect^.on against shoddy performance in this, critic^^l area/^ 

f One could, of course, wite a contract in the absence of a design ^ 
4nd make that* product xhe first requlr^ ' of the 'evaluatbr . ^ This ^ptOcess' 
results in essentially the same benefits ,as that de^scribed aboA^e in the 
fee situation — \4$u^^y, the evaluator is paid to provide a .detailed 
design which is ysj^d as a .'basis for .deciding* whether Che conttaqt 

should be* corltinue^' beybnd the first phase. . Judginent that a design was 

f * ' * * ' * 

unacce()table (and could not, be salvaged wit^ minor.- change^) would reqtiire 



trie 'cenaiaation of a -contract rather tH3n the br^eaking off of negotiations 
as in* the previous 'case, fhis distinction .should not be a problem, however, 
;as long^s orovision for the termination of the contract, if the design 
is unacceptable to the client, is built into the. contract . 

Alternatively, one could' let' an independent ^contract for the design 

* ♦ * • ♦ 

"^hitse. At tJid conclusion of that phase a decision about 'the adequacy of 

the design is. reached. The second contract for th^jpfonduct .of the evaluation 
is thus contingent upo^ satisfactory performance in the first .co^t^act . 
period*, .If the projijctis very large, for example, a 'three year 
^valuation at $15Q,000 a year, it 'sometimes, makes e^nse to let two hx 

even more, design contract^ with' the understanding that the party whoWe 

- / ' • . . \ 

design 'is judged most super.ior will receive the contract for^the evaluation 
itself. ' * . • , . \ 

- t • • c 

3. The» Scope of W6rk ' ' ' ■ < 1 . 

The , second ^ost importanf topio f or negotiation is the -scope of 

\ . - • - ^ • - 

work, for the evaluation. The scope o| wojj^ is 'really the operational 
'Statement of the design. Within it one stipulates, the procedures to be 
•employed, ' the productsxfinstruments, reports, etc.) which can be expected " , 
ffom tjie evaluation, and the timd schedule 'for performance as. well, as ' 

\ \ ' / ^ ' : ' ' ;i ' - ' ' 

"the deadlines for delivery. It' i^ in the ligKt ,qf this scope of work Aat 
one -monitors* the performance of the evaluator in meeting the contract. 

\' ■ * ■ • ■ . . 

The companion paper^by Sanders a^d Nafziger* (1975) deals directly . 

with the information th^ait ought to be included in an. adequate .design 'and \ 

« * * ' 1 ' ' *■ 

scope of work. ' . ** < * 



* «> • V. 

The criteria f dr^ an adequate. d^iga constitute- the basic topix of the 
comp^itibn paper' (Sfhders and ^af^ig^r, 1975). and will not be' dealt with here. 



18 

4^ Issues for Discussxon and Mutual Agreement i 

/- 

There are several issues whi&h, if left^ unresolved during contra^ 

negotiations, can later prove to be sources of gre^t consternation. 

Perhaps the most nettlesome is identifying the respective responsibilities 

of the client and evaluator with' respect to information access * and ^ncjuiry 
t> * 

re^striction. Brickell (1975) provides several excellent examples of 
these kin4s of problems. In i|pme cases the evaluator becoI^^cS-^^ware that 
afi important obj'ecnive was overlooked in- the prepai:ation of tfte contract. 
One of Brickell's example^ deait with a project that irivoived the employ 
ment of parapr/of essionais. The evaluator 's charge in a contract issued 
by the central ^administration of a 'ma^r American city was to examine 
the impact of the program on student learning. But the paraprof essionais 
were directly employed by area sjuperi^ntendents -who had a' veryl^if f erent 
view of the purpose of the program. Brickell describes a meeting with 
these individuals : - * - • / ' ' - • * 

- \ • • ■ . ^ . ■ 

"While meeting with area superintendents tp explain^ the study,. • 
one*bf their spokesmen ogen^d up*" with someth*^ng like- this: 
'Okay, -'you eyaluators*. Lett's get. one thing straight from^ the ^ 
start.. We. have these^garaprof essionais .Here, in these schools 
noto^nly to help kids* learn ""but ta l.ink us to the commurlity/ 
That's why. we have thetn. * That's why,!l/e^re going tt> ke,ep^ them;. 
We're not looking for a^ report about test results* that will • 

. cause any trouble with the inDar4 of educ^tidm downtowns '"^hey^ve 
^ go^ their reasons for giving^ us' the money to 4iire parapro'fes- 
sionals; we've got our^ reasons fpr .taking the money. So no a ^ 

- * 'matter what -you find out" about kids' achiey.ement, we're going ^ 
- ^to .keep oMr paraprof essit^nals . Don't make it difficult.'" (pi 3) 



In th^se circumstances the eVa-^^tor has the iespoAsibility to* 
identify the "hidden ^obj ective^. " He or she -is' nat,^how^yer, required 



^ .^^Portlons of t^ls isection are 'adaptations of Wrifht (1975) 



to investigate the efficacy of the -project in cementing community 
relations. That investigation Vould require new resources ^and a new or 
re^^lsed contract. Indeed this ejcample illustrates the importa^ice of 
the issue or who is involved in negotiation,^ a topic to whiclj we will 
return. At any rate,, it is the client's responsibility, not the" T 
evalilator 's, sto insure that the contract cgyers all of the salient 
areas of inqui^j. The eva-],uator 's role "is only that of bein^ certain 
that failure to address relevant objectives or efforts of a. project 
are- duly noted. ^ ^ , , , * .' ^ ^ 

It is also the client's responSibiltty to insure access to the data 
^tha^ th^ contract requires the evaluator to* collect and/or analyze. *y 
For example', if the" teachers in a given schpol simply refuse to respond 
to'the questionnaires prepared by the 'evaluator , the contracting school 
system must> beat responsib.ili ty for the inevitably flawed report which j 
will result. ^ • 

In the pre'^aration or th^ contract, careful attention must be paid 
.to identifying potential probJLem areas so that, insofar as possible, . ^ 

feoth. -parties enter; the agreement with a. shared unders"fcandin^ of wl^t 

' »* » ' # » ^ 

Utrili occur' and what ^ach'is expected to do t^ facilitate the inquiry. _ ^. 

' . Conti^aet ^negotiations sjiould also ^result in the stimulation of 

the restrictlbns to be placed on the use ^of the re^sulting reports.' The , 

\ r . • . . , • ^ • ^ \ ; • 

contract itself fehoujd. include a precise reference to the intended audiences 
for reports. Faf^l&^mple, sqme^ funding agencies have required t-hat projects 
scheduled tentatively run three years hire . evaluators to conducx 
'*^ummative^' Valuation -studies at the end of 'each year of operation. 



These reports go to the project staff and the funciing agency which uses 
them to identify probl^ areas which need to be corrected, effective 
techniques which should be .continued, aTs Well as io help' the fjunding 
agency justify continuation of the pro'ject. If the evaluator is to 
help .the project straff improve the program or project while it is in 
development, then the failings and flaws need to be carefully examined 

sa that they can determine what corrective actions are required. Yet, , 

^ m ' 

t " ^ . 

docxim^nting. one's inadequacies is seldom the best^approach to use in * 

' ' " ■ - \ 

soliciting funding. . Indeed, it is our view that i"i^ funding agency 

* >* ^ * * . 

wants an objective- evaluation which tnee't;s their own crit^ria,^. then .they 

should contract .directly with an evaluator. Forcing a ^unded- client to 

hire an evaluator whose reports go to .the funding -agency is rathea? Tike 

/ - • I * 

asking the infested .'Country to provide a per dieit f 6r fone's' 'resident ' ; 

Spies. ' ^ ^ . . ^- ' ' v . 

The negotiations should also serve as an opportunity to explore ' 
potential conflicts of interest* Is the evaluator currently employed 
by a project which is competing for funds from^the same source? Is 
there any reward likely for 'the evalUator if th*e. report or the project 
is favorable or *unf avorable? vThe eyaluator who has a spouse on the 
project staff is Likely to influenced in .report preparation .by what 
would be most Advantageous, for the spause.'«' Similarly, if the external 
evaluator is guaranteed continued employment if the project" is refunded 
and the evalu^itioh report will bie .crucial ^n that^^dec-ision, then the 
evaluator 's own financial self-interest promotes favorable results. 
Means of^avbiding such QoufLlct sitpuations should be exploded. 



* » Another '^issue that sometinies arises is the "piggy backing" o^f the 
evaluator Vs' research interests. For example, an eyaluato'r has a 



continuing interest in personality research and wishes to' admiiii'St^r * ' ^ 
some new personality assessment measures to the teachers involved* in 
an evaluation project. >Is it' proper^ for him^ or her to do so., If such . ' 
evidertce would be unrelated ; • or only tangentially related, to the 
client's evaluation concerns? It seems ^ to the authors that such an , 
activity can be legitimate, but only if 'it his been mutually, agreed ^ 
upon. The best time to Consider the issue is, during negotiations so - * 
that the contract can stipulate provisions and restrictions . ■ . 

In 'the same veinj sometimes ^ the evaluator can use instruments v 
developed under one contract in another very similar evaluation. Is / 
it legitimate? Yes, rt is, unless the previous contract specifies 
otherwise '(which would seem appropriate only where 'instruments *have 
commercial potential or other characteristics tha^ would be of direct . 
benefit to the client.) In the abS^nc^ of such contract provision, the 
evaluator is not restricted from using the instrument elsewhere. • ^ ' > 

The publication rights Issue .goes be^^ond simply re-using instruments . 
When the project is completed the evaluator may wish to publish the results 
in a professional journal and would' probably be wlllinig to disguise 
the context of the study so /as presery^ the Ki'lient confidentiality. 
The client, however, ,may wish -to avoid or reserve to a later time any 
publication of the findings* Again, -mutual agreem^pit .concerning^ the 
rights of publication aji^ri^ved ^t during thV^ negotiations phase would 
alleviate or ^eliminate problems^at a more critical junotur^e. 



\ 
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Stake (1975) recently compiled the ^ responses of several prominemt 
•researphers and evaluators to seven issues involved in .commissioning an evaluation 
study. The. reader is referred to that document for an enlightening discussion 
.of ''piggy backing" and' several other concerns raised here as well-^ . . ^ 



.Sometimes, the cliajat must dr. choafees tp report Be^ul^ts directly 



to some audience, for exat^i>le' t,h^ ^funding agency. Jhe c|Lient may, • . 
abstract or modify the report submipted,^ by the evaluation contractor 
in-order to aqcompTish this end. \)ceksionally.i.,:not often, the evalu'ator' 
^perceives that^repOrt as misleading or , inaccurate a-I .The ^^evalua'tor ill 
the authors' vtew then has a professional, and *a mbt'al,' respon3ibilit7 



to resfjond to this .misinterpretation or misuse:^6£ liis^ or her work..,. For 
exai];iple, an unscrupulous project director reporjtjs^ a yocatipual training 
program as an' unqualified' success-/. The report .specif i^s' th^t 100 tp^rcmt 
of the trainees were employed in full-time Jobs .in; thair. c,hosren.f ield' at^ 



< - 



the conclusion or the programv - The^evaluaCtor ktiows , th^r "65 percent ^ 

of the graduates were dismissed from these».iobs for incompetence' within ! 

two months" after being hir^d. 'Since other studTents j^ay be s;lLmtlai:ly /> 

Victimized^* since other agencies may be misled *iiito- attempting' to'-* ' 

implement the same 'training program, thB ev^laa,tor^ had no a Interna tive * *. ^ 

but to expose the fraudulent' us^ 'of the dita-^ ^^^he means ^or rebutta^ ;. 

could vary from a letter writ*t^n to t\ie rfecxpiefits of - -the retiort to • ^ 

an appeal to the ethics committee oi so"me prol'e'ss'jLonaCl association or 

a public pres^ conrer.ence. • ' '. *' ^ 

' . » *. - • . " ^ ^ . ' * • : * 

A related situation i§ the non-use of -evaXuatiori findings*- 'The', ' . \ 

evaluator has.^o right to JL^si^t that his or ^'her findings ''bel.^e sole 

basis for the client's decisions." Indeed ratioiftal decisio'n* maki'ng - • 

» ' ' • J, 

demands^ that' all of the information apd^evidende available, not just 

phat which is included in, the evaluation report , be' examined 'and ^weighed 

in the decision making process. Yet it can ha{>pen that ar. cYiti(;al' ^ 



evaWation findings is ignored, either del'ib'er^tely or 'through 'ciischance, 
by tho^e reaching a particular decision. Depending on the consequences 
jov chat non-use the evaluator may feel obligated to. respond. For 
examp^le, the evaluator found evidence to.saggest that the use of a 

f 

particular social, studies cu^rriculum ^exacerbated racial discord among 
the students.. The decision is made to continue the use of the curriculum 

because the ^deci^iph makers did not, receive or did not take the .time 

' ' ^ [ / ' " ' J 

to read the evaluator 's' report. The ^valuator must take stepis to ifiform 

these individuals of .the data. Subsequenbly, should the decision remain, 

unalcered, the evaluatpr may feel compelled to addrfess ^'more public' 

.•f*orum. . ■ ^ ' , 

/ The problem hai another side to-i.t. Sometimes the eyaluator errs 

and includes inadcurate information in a rej^ort which is circulated to 

a widef -audience. . The client is now. placed in a defensive postur^. 

;7he report, is wrong and^must be corr,ected. Yet :in correcting what-may 

be minor factual errors, the client may discredit a report which is 

by arjd large .favorable. He or she is placed in an extremely unt-eilable 

Dosition. I , \ " ' , • ' ■ ' 

The most unfortunate aspect of these situations i^ the fact that 
. \ ' ^ . - * ' • ' * ' ' . , 

they ^re* usually sO -easily' avoided. Ir^ most .instances the misuse or 
noA-use o*f 'data -anises not from a lack of scruple,* but from a laak of ' 

understanding. If the cohtract 'included provisions^ for the review of 

, ^ ■ ' ' » ' , ' * ' • , ' ^ ' . ' * . ' 

reporjts by both'part^ies prior to rjelease\and for the adjudication of 

s , ' ' ^ " ^ 

d^.spuces', most ptoblems^'of "this '.nature woul^i probably' be eliminated. 

In the instance of , the- unscrupulous vocational training program 

' * ' ' ^ > / 

proj'fefct "director', a conference with ^ the' evaluatjor might result in an 



iapasse. The director jsees an opportunity to toake hay while the sun'**" 
shines. Th^ evaluator, one hopes, will not tolerate such fraudysbut 

; ■ 

will make-'very effort to expose it. In the other examples used, however, 
it is doubtful that disagreement would persist\ Reasonable individuals 
will find acceptable resolutions. Thus a provision .for prior review 
to correct factual errors and examine differences in interpretation of 
the evidence would substantially ameliorate the situation. 

An agreed upon adjudication process would also be.^benef icial. 
There are many means that one can employ to achieve the resolution of^B 
disputes. A binding or a non-binding arbitration clause could be 
inserted in the contract. The arbiter could be a professional member 
of the American Arbitration. Association. Alternatively, an agreement' 
could be reached on the use of an individual arbiter or a panel of 
irbitraters chosen from the field of education.* The panel offers the 
advantage of permitting the representation of multiple perspectives, 
e.g., evaluation, administration, and the relevant instructional 
•disciplines. The panei should , -however , consis^t^ of an odd number of 
individuals in or^er to avoid ^tie votes. At least One member of the \ 
panel shoMld be technically competent in the appropriate evaluation 
methodologies, i.e., statistics, measurement, d'esign, ethnography, 
observational systems, etc'. If that is not possible, the panel should 
have acc el's to -expert testimotly. ' ♦ ' * . 

Even in the absence of a binding arb4.tration clause, the contractual 
^agreement: for an 'adj udication process could serve a valuable purpose. 
In the case of conflicting 'interpretations of the data, tqr example,. , 



ic could serve to legitimate or discredit particular courses of actTon. 
Take 'the case of the social studies curriculum. If' the client feels 
that the data on racial discord was inconclusive or unreliable and for 
that reason should be deleted from the report, arbitration might fail 
to resolve the differences of the client and the evaluator concerning 
the worth of the data but might suggest that bo^th points of view ought 
to be included in the report. It could thus discourage and discredit ' 
the use of a more public forum. 

The" same approach to conflict Resolution ^ould /J)e utilized'in 

^ — _ - ' t 

instances where the clienf* wishes to terminate a ^contract, for excunple, 
on"the grounds of nop-p^r^ormance , or the evaluator wishes to terminate, 
for example, on'"the grounds that the. client has nQ^: met contract cohdi'tions 
concerning data access. Termination j proc-edupres should^falso be a -matter 
of contractual-. agreement, \ . .s, 

' Regariless of the means cnbsenn and iri^espect j?ve of the perceived 
likelihood that it will be netessary, 'ifc is recoiflnended* that^.an adjudication 
process be negotiated and included in th^ "Contract . The mere availability 
of the process often serves to promote betjier communication and more 
facile resolution of disagreemjents -betwe&n clieint and con^trac tor . j". 

Of course the contract should also ^tipulate the basis of payment. 
Not only the total* sum og; the contract, price, but the payment schedule 
needs to b^ clear. Some funding agencies hava^ restrictions concerning what 
equipment costs can be legitimately paid, fpr -example, br the amount ^ 
that can be enlarged for indirect costs. It is wish to include these 
restrictions in the evaluation contract to avoid confusion, Ip is 



also useful to ir.clude in the contract specifications for 
appropriate remuneration in the event the contract is terminated • 

'x ^ 

5. Parties Involved in Negotiatrion * V • * , . 

One common source of difficulty in the conduct of 'evaluations is 

tlie"fact that significant individuals or -groups were not included in 

the negotiation of the, contract . An excellent case in point is ^the one 

related by Brickell and quoted earlier iri thi^ .paper concerning ^^e . 

area superintendents who had different expectations of the par^prof essional. 

project than did the central admi?nistration, , "Had the central, administra- 

tion thought-to involve the gSfea superintendent's in Xhe negotiation of _ 

:# • , ' ■ y 

the*contract it is highly- probable that a more comprehensive evaluation 
would hav\B occurred. Certainly?^ the evalu^tor's role would have been - 
clearer and lessv-f raught id.£h conflicts As a general rule all who' will 
be affected by the evaluatix)n have the^ight to be involved in the ' , 
negotiations — at least "^by representation^ : Too often, for example, we 
expect teachers and students to comply completely with an investigation 
wlthput evfer involving them in decisions about what .data will be collected, 

^ • \ " 

fr^om whom, in what manner, .or when. It does not' seem unreasonable to 

suggest ^^Chat at a minimum a representative of each^group could be 

^ 7 " . ' ' ! * - ^ ~ - ... 

. ' . '» ' 

involved in.Xhe negotiations and report back to thd^e he or* she representsi^ 

^•^ The preceding argument should not be taken to mean that these* groups 

ate entitled to.be involved^in all aspects *of the . negotiations For 

example, since the argument is tenuous at best that the.tedchers and 

3tiidents are affected by the, annual salary made\by an evaluatqr, they 

are not ^^ntitled' to that i.nformation nor to involvement' in negotiation 

concerning* salary ' or budget. It niay be that some day all publicly supported* 



races, sal 



cotitraccs, salaries and bud'gets will be open to public v.iew. Road 
concractors, public works departments, haspitals, ecc, and their 
employees would then all be subjected to t"be same scrutiny, In\ that 
case the evaliiation contractor's records,HhoUld ^Iso .be open, ^ Until ^ 
that day arrives, however, singling out? the evaluation ^pontrac tor is 
simply discrimination, , • ^ ^ 

• ■ ■ ■' " ■ ■ 

Moreover', one must^bear in mind that' invclvement in the negotiations 
does not equal veto power, Ihe primary negotiators are usually the funding 
agency and the direct client. Thus, if 'the state^department of education 
provides' money to a school system for a,/ particular evaluation project. 



•the state ' department and the legally constituted authorities of. the' ' 
local system are the .principal negotiators with the evaluation contract 
The involvement of . principals, teachers, studenjis, and citizens is at y 
a secondary level' and* limited to those aspects of the contract which 
^iirectly affect t^^em, . ' ' »^ , , 

The involvement of. these groups is in keeping with^ the fundamental 
valines of a democrajtic society. It also tends' to maximize communication 
and* reduce tlie threat of non-cqjoperation- on the part of 'those who ^either 
don^t understand what is going on or^ are alienated by virtue of. their- 
exclu'sion srom the decision making process, 

. 6, ^ Procedures for -Terminating NegQ(tiations , . - 

Con^der -the following situation. An evaluation contractor ha^ b^en 
selected -either on a sole source basis or by means o£^ an RFP, Negotiations 
have been going on for two months, the ^project will be implemented within 

" /' ■ yi ' • - ; . •' . ■ " ~ 

'Che nexc" four months. Time is getting short:. Does" the client brea.k of { 

• ■ , ' / - - , - \ 

.negotiations and b^giu bargai,tiing with their sec^bnd chance? ^ It all depe^nds 



It depends on whether the negotiations are ^progressing smoothly 



and^ a. contract can be expected soon* It depends on'how much lead time * \ 

- ' - ~ ' I ' ' ' '* . • 

berore implementation the -evaluation will require. It depends on how 

^ ' i 

s^uch money has been invested thus far in the p^;:6cess, ^and what it is expected 

to ,cost before^a contract is consummated. Ijt also depends on wl^ ether 

the C/ontractor aria* clieirt think ^that it is worth the expenditure^ of ''more 

** 

time and energy to pursue the negotiations. 

■ ' / ■ 

Neither, the client nor ,evaluator can force negotiations ^to a successful , 
conciusioi^. If irreconcilable differences exist, the only intelligent 
course of action is to' agree to terminate negotiations. The wisest 
way tb proceed might be for all the parties involved to set some resource 
and time limitations on what they will invest in^the negotiation. 



CHECKLIST: Section Six: Criteria for Us e in 
^ " ' Negotiating fhe Contract 



!• * If an experimental design is appropriate, 
arQ^ the negotiations ^chec^iiled to occur in - 
time to permit eyjaluator involvement prior 
to the assignment of the treatment to.the"^ 
experimental units ? 



Check one for each item 



Yes 



No 



2. If a time fraihe approafeh^to tl^e evaluation 
is appropriate, ate the^hegotiaticns ^ 
scheduled to, occur in tim6 td per^nit the 
' ' collection of data {concerning the status 
of those ^vho are to participate) well in 

advance of the introduction of the treatment? 



If necessary, are the^ negotiations .schedule 
to occur in time to permit the development, 
trydut, analysis, and revision of instruments 
prior to their first intended xise? 



Havfe adequate provisions' been made .for the 
V development .of an evaluation design, through 
'^^ne of the follo\\angr preparation of design 

for s'et fee; preparation of design as the 
* first phase of the contract; or preparation 
of design undei: an initial contract, witka 
separate contract for implementing the design. 



5. Does the negotiated Scope of Work include: 

^i; * the procedures-to he employed by *the 
' [ evaluator and £t time SQhedule^or their 
• > performathce ? 



adequate descriptions of the'.pro ducts to 
be expected Jfrom the evaluation and 
dea dlines for their^deliverv ? 



V 



V 



c, criteria for [judging thai the ycoptract 
, ias been fulfilled? ' 



3 



Haye the negotiations dealt with the 
respective responsibilities of th^ client 
and contractor concerning: * ^* . 

t 

a. the identification of the ot)jeQ.tvve8. of 
- the inquiry ? ' ^ 



h. data access-? 



\ 
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Check one for each item 



«r 6 I 

1 



Are th^ audiences'of the evaluation 
^eport(s) iderifjfied? < ^ 



8. Have negotiations cdncerping potential ' - 
conflicts of interest takgn place? 

9. ' Has the issue of "piggybacking" other. 

i research interests been discussed? | 

10. Have^tlie copyright arid publicatioja . ^ 
rights issues been thoroughly 
explored and mutually agreed upon? 



11. .Has an agreement concerning 'the ' 

' rights of prior^leview of reports beei\^ 
reached,? , • ' ' - . 



12. H^s an adjudication procedure been 
estabUshed? - 



13. Has a b^sis for payment hepn 
established ? 



14. Have thp pftmary negotiators been 
" identified?;^ ^ ' 




Have those^whb will be ^atfected by the , 
valuation been invited to participate-^ 
at a secondary level in, th^- negotiations, 
at least by representation?* , 



16* Has a time and resource limit for. 

negotiations be6n set by mutuallagreemeet?' 



17. If the negotiations have not been success- 
fully c<jncluded.with the time and resource 
limits set, has a second potential < 
contractor been identified? ; (Refer to. . 

; checklist? section ""three. ) * 



- Yes^ 



.\ No 



SUMMARY TO SECTION.SIX : 



Bdsed OH questions 1-17 above have negotiations 
been sufficiently successf&l to warrant letting a • 
contract or contracts for accomplishment of the 



ev^^iatibn design and conduct of the ^valuation ?' 



le ^aiuj 
3tu^ to 



(If yep, sign the contract; if no, ire 
four and use tpTwid^ another: contractor. ) 

1 *i it , 



section 



1 


• i ' 
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• 
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.1 . * 



; A,s^me Ghat the negotiations have been successfully' Vonciuded and 

that a.^contracfc to conduct' the. evaluatioa exist3- ^ If- the^ ^egbtiati^jh^ . ^ ^; 
* \ . ^ 'I ' , ' ' • ' T' ' » ^ • 

.have been well conducted, wV doets ,one\need, to mon±t:or the coatract?'* 

, There are tw'o basic rearsons.for monitoring, '^he first. 4s .EqHdehti^y ' .-^ - 
-'unant^icipatedeverits which /Will'.af feet W capacity "of the cl'ient or J . 
eValuatpr to' qompllr fully with the contract so -that" renego {fiction, of . , .* , 
the contract, if necessary, -can, occur, ffhe second reason for monitoring 
is to identify proitptly a|-.^s .of dissa.tik action oa the part of either 
the '.client' dr the co^tl;actor jso thae:' 'small problems do not grow into a " ** ' 
, - br-each-pf contract isituatiori; and to fapUitate mutual agreement to 
• ' terminate the^c;>atract if -that, seem^' co be th6"^best course of actipn. 

s The Tools of Monitoring ' .? .• ' : 

, - 'There- ar,6^ ifwo tools- that- permi? the« client to. "idohitdr the -performance v 
,. of Che -contractor'.. .'The first ^6 the scope- of wor k. section' <if thi contract.- 
^ TCe. second is the criteria^ est"ahiished for 'the procedur.e^^^nd. tije. pro'ciucts 
^ specified in the cotftract. ^ Th§ latter tool 'is ■the-topip -f or tUe. .companion 
•;papeT (Sandets- and Naf ziger ;.1975)- andf-WLlL^not bQ'd«g;it ' with-^hfere . • " ' 
^» /Other th^n the qualitative judgments *^to '>e. made, 'th'e^ are-four ' 
basics questions one can ask.or th$ <'ev^luatar>''s perforinarfcef' ► ' * 

1. 'JDid, t;he evaluator use the procedures 'thdt were agreed upon? " 

.2. Did the evaluator use Ithe procedures within* the time schedule • 
f^agreed upon? - * . * \ * . 

3;. Did the evaluator deliver^ thp products agreeci upon?*' 

4, Were 'those- products delivered by the agreeci 'upon deadlines? * 



G.. Standards atid Brjocedures' f or -Monitorini^ \ ' ' 1 . ' > 

^ ' an Evalu^tloii^CoQtract - • ^ ^ . . 
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G.. Standards atid Briocedures" for •Monitoring 
. . • ^ ' an Evalu^tlot^ Contract - - 

• ' \. ' • ^ . • ^ ^ 

^ ; As^me tihat the negotiations have been success^uil/ cbnciuded and 

that a. contra^ to' conduct" the. evaluation, exists.'-' If-" the^Ug^tiatifth^ . 

.have been well condected, wV do€is',one>eed. to monfrtor the contract?'' •• 

There are tw-o basic rearsons.for monitoring., '^he fiyst,4s .tgHdehtiCy ' 

•'unant^icipated everlts which .Will'.af. feet "^the capacity 'of th'fe ciient or/' 

eVaiuatpr 'td Qompljr 'fully with the contract so' that" reaegodi^tion.'of , 

tlie contract, if^ necessary, -can. occur, fehe second reason for monitoring 

is tb identify promptly a|eas ,9'f' dissatiLf'action ou the part of either 

" .. ' , ' ' '• \ « 1* " ' ' * •'.'*'(" 

thelclienf Or the coAtr;actof iso thats'-sinall "problems do not.' grow into a " "* 

br-each.of contract* isituation,' and to fapUitate mutual agreement to 

terminate the^^cwttract if -that, seem^' to be thfe'^best course of actipti. 

^ The Toiols of Monitoring ' > ' ' « ; / " 

^ -'^here^ar^^tfwo tools^ that.penni; the^ client to'niohitir the -performance 
of the -contractor I .'The first the scope-of work.sectionK?f thd contract 
^he. second is t^ie criteria^ Qstaiiished for 'the procecjur.ejs^^nd. tlje. pro'cfucts 
specified in the coiftract. Th§ latter tool 'is ■the'topip'for tlie companion 

- ' .•" • - ' • V- ' ' •''>' '•' ' ' . ■ 

;paper (Sandecs- and Nafziger ;.1975)- andfMllvnot d^^fe' wlth-jhere. • 

/ Other than thQ qualitative^^ judgments \o '^>e, made,' thei^ are-four ' 

basio questions one can ask.or th^.ev^luator>''s perfor^'ante-^ ' • 

1. ^JDid^^he evaluator use the procedures 'thdt were agreed upon? " 

^ ?. Did the^ evaluator use Ithe procedures within* the time schedule • 
^agreed upon? • ^ . ' \ ."^ ' , * • . * . 

a.v Did the eValuator deliver^ th^ prodiXcts agreed upon?'* ' * 

4. Were ^those^ products delivered by the agreed upon deadlines? * 



^ . .. . . - , * 



- '.^"'^i'^'^"^"^ Impossi^le/to'monkSr. t^^^corftract, to establish contfract ^ W 

or preach.;. It'^iS ' iiTi^^Fa^-'no wib" iiJuafiin^fj^ " }" ' •' 



•4 



•. . . -'2. " Proce aiurfes for 'Renegotiation of 'the' Contract' " - "' 
' ■ (' ' ^ ' '' ■ . • ■ , ■ ■ : • . ■• " . 

. " qftenieVnnoi th6. evalaat'ioru wiU not be. accomplished" striQ,bly: 

i irf accord wita the" "contract. .Xhtre always^ seems to be some uneifpected, 

- -y ■., . ^vehe fhat /oroes'm.odif icaejion of ^ the otlgin^l* intent ^f-,- for exam^l-e, , 

• three of si3c- teachers who; ate Wng a new .math, ctirricala^^ 

^ , . one,.reascfn or aAoth^t; it ±i cl'eai?"that some change in, 'tl>e^ evaluation-V 

-Resign is in order. ' ' ',*'■"' '''- ' \- ■"• A '\ 

: . • . ■ Whenever circumstances/change f rom those -anticipate in 'the coiiracj: " 

-the^.ciient. and jthe contractor need to; consider whether V^negtrtiatfion .is \ 

-• ■ ■ r.equir.ed: Both 0a}:'ties |hould 'examine' the spope 6f work cbntkiued in the I 

w , ■ '• '. - ^ 

-<* • .« • ' . " ■ , ~ ' 

contract and determine, which elements can.still.be accomplished a'nd which ' 

; ' ' . . ''5 . 

• .are nb longeT^sible.' Ad justoent tan then b^e'made. T^e .tinlelines.may 

,2 , ■,'\eed to be a-l^red. Mo.re^ -or . legsrmeney niay be •ap-propr^atiTgiven^the' 

' ■ ~ ' * * ■' • *: ' ' ' 

• Chang ejL- si taktion.- In the event -that /the; client a4id contraaortit disagree , 

/;.-/^'^abp'ut the kind or -^tent of vthe Wiiica;ti1ns required^^ thV'Idj-udication.' - 
\" ^ P.>^°<^ess' stipulated -in. the cOntra<it?'should used." ~ " -• 

Given,. thq cons traints:^f; the re^l world' of'; t^-^^^^ flfexibl-iXty. ■ 

, / jLs .e^sehtial >or hotl parties-. - On^' aljy-feu.st' on occasion concede/that ■ 
^. •■ fate,;has -d^cre^. that no .evaluatioa^svto opJjpf; '^Thrqu"gh the faatr 0 ' 
' -no ®,ne/xhe^ityation is^^hd^eles's^^^ A^m^ansfoV termination 

•\.;7 *.*^.>. * ;> ; 

o^- the 'tpnVrac/i^v^lTea'^sbfot^ld 'iJe' used. - - T; ^ ' ^ ^ " > ' . 
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3. ^rocedur-es for Termination of- the .Ctfhtrac b ' ■\ ' r- ' ' ' ' 

^- When both parties agree, that theiyontrac;: sho»lld'b> terminated,- they'-, . 




ought to r^fer, t9 agreemen,ts -contained in t^e concfatt ooncerningja^prppi^ate ' 
^ remuneration atidl clp$e-out_ procedures. ; A Tjrief rfeport' i4entifyin« ;the . ' ' r J 

4 f i- - '" \ t ^ '<■'•.*,' ''' { -' , 

. y^asoas/f or •fcermination sh6liid*be prep'are^^antl :^f iled ! -^If .either ,^arty ' • ■ 

ddes not^ wish .to ' t ermimte the''contract'-pr , if a dlspi^t-e' ^iscs- cdncerni'ng '*' 
what would constitute just recompense or •c6qcernini-;some othfer factor ' " 
Mt is approprl:itdl:o--:taple3nent .the^adjud'lciitiQjl p-roces^ 'agteed ^upoh " 



4 



in tftec;^rit^ract\ 

'3^SlJ!^V^>i^^ vdfyrjsiatxxTxsO^ pioihtrat which to. c:onsidfet "termination bfr.. 
.^he^>cotitractr,ii^3:3^ ti^e design- i^base, Unless, 'th'^ fee or 



-—separate ppntract appr-ciibh^^s used., .'The other points need to be buii 

• ~'* ' '■■ • •' '^NsZ^ 'l- . i' ' ' - / 

,-.-_into t:hevcoiitract: , For tte' B^,il(2fes^;^.the ^Wline^.ton^tit^^ 

• at which « one 9an famine 'tontra-c't- ^M^l-ii^e^^ '.■D&temi^ing wh$th/r -the' , ^ ■ 
i • evglaation^ is.' fQliowing -Ihe agreed 

\ pi;Qbl6iiiatic-. The us> , o^' a "mirestone".*app-roach has b'een. us^eH'- with somV.v 
' • • -' * ' " ' ' - '. ^*.' " •*'.>.• ^ ■ , ' y I , . ; 

. Wtfessv" yithiji t|i^ contrabt on^ stipulates a. specific ^o-yaj; in t^e^by '.- 

[ -whJcH cei^tafh events Are_ to .have -occufereL .^At'each of -tliese time , point's 

/■^^raB--64^^ent^^ review progress' tb see whether 

. things- a^q%n/ta?]^^^2or example,- t^a 'contract .for the evaluation of, a' 

-^cifence-'Prigram-'iiay cali^ the- admlnis'trafcioprdf-.^. pretest to'v^'tudents. ^ ' 

x/tnterviews .^iich each experimental- teacl^er IndU-^cO'st comparison .-with 

; majf n.sarr^^ havB Xakea place by December. 1, 



On or, shoi;hy attor tha't daxT^^ient knd 'caati>ac'tor'' confer to examine 

. ' delays,^ overexpenditure^, and. the like^ ar^ th^ r^asdns for thejn. If the. 

' '''' / ' • * ' ^ . */. ->'^^" ' '\. ; ' * • * 

probl.etDs are ^serious enough, one considers^'the' possibility/ of . tenecoti^^ " 

nation ♦ • V • . ' - ^ ' : ^ ^ \ * : - 



or, of. termination'/ 
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, ^ Detepaining which breaches -of the contract are aerfous eiiough to 

* • . ' ' * ' • , - . * : . < ^ • . : 

m^rit conslderingf terminatioii is dependent on the situation and the 
^% ^ ' ' ' . • 

, individuals! 'Certaiti cases are .so extreme that solutions are'ea^y*. The »' 
la^t' t;^c}i6r inxfetvlew was^ condu,cted on December' 2 because .the -evaluatoi? 
paught the flu and ted^to res'chedule. 'No one in th^ir right min.d would 
Consider dissolution of\ the contract 9ti;:th^t ground. On' the otti,er ' 

* \ ' * " ' • . ■ . ^' ; ' ' . ' . ^ 

.side,- Xf no ♦one. has sem or he^rd of the ^valuato>r* since the 'day the^ * 
. contruc^ wa.s signed, ^it *is relatively** simp iV to* deci4^ that* breach pf " 
^ crontract haa occurred,- and to-te^inate.' * Usually, -however,* the situation ' 

is not as clear, cut. Judgn(ent and ^fehe irfteili&ept application lof 
-experience is requird^. \ ^ '^d ' « [ ;• ' ' 

^ P.^rhapS the^ best -^dvic^ -onk can of f «r fis that tha communication of • ; ' 
d^^a^is^action rnt^sj: be direct aVid open and .documented very, carefully.. ^ \ 
If the contract'ot^ ffeels that ^rk ,fc^ whioii- th'e. client is responsible is . 
not beiftg performed ade^qiiateLy,! and ^ that this failure is adversely ..affecting . 



the evaluation: -effort, .that. ishould^^^disfcussed between the parties* . , . 
The results ot that meetifig ' shauld be-notfed in a memora,ndum 5r letter* ^ 
If the client' belieyes^'thal; the'evaluatoil' is failing" to per^orm'Ms 



responsibilities, again this situation s 

. » * ** • • 

Almost * any- linilateral .atjtfetopt to ^termin^ 
prior ^'^pressio^ of dissatisfaction is 
to a debilitating -and p*to>tracted disputa. 



lould'^be discussiad and documertted^ 
e t^e contract without some » 



oihardy; ,It is ^n invitation 



, Onej great 'ad^Tantage of an agreed liion ad^j^dicaCiion p;:ocedure is the 

' ^ i ■ ' ' ' y * * * 

fact th&t,one ds. reminded of what wilt' ivre4uired to S5tis|y-th^e irifprmat iotji 

demand^' of ^the- ^^rbiter (s) • Xn/preparinf-^thafr- Informaticm' one' is'af'tfen ^ble/t 

' ' " ^ I ^ * * ' * * ' ' ' f - ' ' ' \ " \ 

Co tracfc^own the jreal' s?)urce of the pYfoblem and a^Iicably,.t:enego^iate .x>r . \ 

terminate the' qontract rathftj-thanv-'et^t^.a conc^en'tictu^ fray. ' ■ ' 



CfifEeKLIST; Seetioa Sevea: Criteria for Use in 
, ■ Monitoring the Contract 



I. H^e^proc^ures for moaitoring.the progress of 

the work beea. agreed upon, i.e. , have ^ v ;^ 
miles|:oae review poiats aad product deadlines' 
been clear ly^set? . ^ * c • 



Check one tor each item 



Yes 



No 



2. 



/aatibipatejJ ip the contract, has a reaegotiatlon 



citcumstsiiices are different than -those 



occurred'? 



Jf /renegotiation was not possible, has a mutually 
satisfactory agreement to tertiinate the cx)ntract. 
Jeep reached ? \ ^ 



4 , \ '■ 



/If a dispute §xl%ts, have the two parties iiiet.tb 
attempt a resolution? ^ Have the results of thkt 
meeting been documented? 



If the ^dispute cannot be resolved among the 
paAies to the contract; has the adjudication 
process been invoked? 




SUMIVTARY to SECTIOI^ SEVEH ;. 



Bgised on Questions 1-5 above, is the conduct 
evaluation proceeding. in a mutually satisfactory 



fashion ? (Ifyps, proceed to^cbmpletion of t|ie 



equation; if no, modify pr terminate contraJrt. 
^asjiece^ssary. ) . ' . ^ 



1 



\ 



^ Application of .the > Standards and Procedures *^ 

^ ' ' to gafmple Contracts ' - ^ . ' ' 

. • -^.'i ■ ■. . 1 • ■■ ; ,■ ■■■ ■ ■ 

In order to illustrate ho^ the standards and procedures proposed itv this, 
peper -might prove useful, three hypothetical" contracts have been const'rupted.' 
The first; is a' contract for the design ^of an evaluation*' The second is a 
comprehensive' contract for tKe conduct' of that same evaluation;. Each of 
'the^e contract^ ,has h^een formally, drawn with a great many details provided * 
'in the 'contract to illustrate how coiiq)l6teness anc} precis itiri^^ght be ' 
obtained. ' ' . ' " " , ^ " 

• , The third cpntract i^ far less formal- and complete.^ It illustrate^ ^ 

the minimum Content' of. a contract* Since many contractual areas are not 
* f * , . ; 

dealt with, a>l<^ument of this kind assumes a higfi degree of trrust and . 

confidence .between thVtwo^part'ies* Each relies on -the other 's/p^ofes^ional 

' ^' ' . ' ' ■ ' 
^ sen^e of responsi^jilitjr. 
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CONTRACT . ' « 
. Foii.AN 
. ^VALUATION DESIGN 



This agreement entered into as of this 1st day of June, 1975, by and^lj^tween the 
School district of Metropol, hereinafter referred to' as th'e "diWtricf ' aiid 



^* ^ Evaluatoi^'s Anonymous i Inc, 

H9^al Fontaine Road 
Junfeau, Alaska ^ . ' 

hereinafter referred to as the ^'contractor. A : ' 




* ^ ,. Whereat-, the schoofdistrict of Metropol, proposes to ^yaluate the Pre-, 
•school eoncepf Development Program whichas.to be installed iageveral^kfnder- ' 
garteii. classrooms asxof' September 7,' 197^5; and ^ T.: ' . ' ' . 

Whejffe^, the school district'desires to avahitself of .the sei^vices of a 
contractor experienced and quaim'e(J in this field; and • ' * ' ' ' ' V' ^ ' 

Whereas, the contractor- is willing ,to undertake this endeavor- * • ' " v 
^ Therefore, the parties do mutually agree as follows-' ^ - - ' 

1. . Scope of Services • V • . ^ 

• The contraotor agrees to design- an evaluation 
Development Program w^c^wiil provide answers to the 

A. How is ^s program being Impleinented in the c-lassito^s in wthich- 
it is used.? / " " . . . • 

Are the Children in, class roott^s- in .which this program is used " 
mastering its objecttves ? • ' . 

C: ,Do.the children in classrooms using this -program achievl-a higher 

• « ■ ' - ' • '" ' , ° ■ 

m ean score on an appropriate ^xamin^on than 'children -in other 

'kindergarten "classrooms ? ' - ? • - ' ^ . ■ ~ / 

\ 

p. , Do the teachers using this program view it as a valuable adjunct^jo . 
• ^eir instructiohal effort? - ' « ^ \ ' . 

In orJer to accomplish this evaluation, the cou'tf a'Q|or agrees b' present 
to the Boatd of Education a , proposed eyaldation'design'ho later than-June 15, "1975!' ■ 
The evaluation de^i^Vs to|/incorporate' the following p'r ov^^ h " " 

• ■ '■ ' ' • '^'t,' ' , . - ' 

' A • - XT - f'H ' . • ' \'' 

' -A. No mdre th^ 6 of ISavail^le classrooms are to receive the~" -.-C 



B. 



X. B'.*- The selection of olaasrooms foivparticipatiou misrfee sen^inVe'to .'^ 
the multiple. ethnicity of Mpferopol Ind stili^pennit- valid cfpmJ)ariSQ^ 



/ 



V 
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,.of. participating iu^ non-parti cipaiting^c las srooxhs,, • ^ , ^'^ * ^ 
<C'. Any^in^tnjm^tjts to o^'oreated, /adapted, or AiSjed in tbis evaluation.' 
.\ ^ . ' are;td be ;specJ:Qerf in'the H^sign^ Moreover, the frequency of use 
^ ' and intetidey're€j)ondents^ are to be similarly speeijBecj; > ^ * * • ' 

D. , .Any sj;atistlpaf anajjrse^ 
' >v . - : \ev^luati9n a^e tobe'^p^oified^in th^^ * ^ 

- \ ' . ^Pates are to be spe5ifi^d fgr tbe^ubtnissici^f .an interirb^^rrid a ' 



1fhi§l*rep6rt. 





The pi:opos6d evaluation.'design wifl be reviewed by a coriiinittee selected 
6y the Spioel Boai^d ^ ' 

menib^f of thp Board jc^Education 
^T^i.e Superintendent' of Schools • 
One elementary school-principal' 
One IciiKiergarten teacher ^ 

One^ax:em^s^f^ . ^ " ' 

^ ' ' ; The feyiew is totake place on, or before June 18, 1975f' Based upon 'the 
review of the proposed ^design,' the committee will make one 6f three decisions':, 
for and in behalf of the district: ' (1) to accept the design as is and issue a contract 
to the contractor to proceed with the evaluation in terms specified within the ^ 
design; (2) to list required modifications in the design to serve as a basis for/ 
negotiation *of the design wlth^the dontractor; or (3) to reject the design and 
terminate further involvement of the contractor in the" evaluation. .In the event 
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. ^the •second. dfecisip^ is rendered, negotiations' are ,tp begi^ oir June 20, 1975 at a 
. mfe^ting to be held in jthe' school dist:^ict'office itt ^ettopol. [ They are to^conclude ■ ' 
^<iii^an:a^re4inen^^^oCeeiJjWfth' the i^pl^^n^tion of a negotiated evaluatioi^ • *^ 
design §pecifieji by contract pr to ferininatef nego^ations' no. later than July 1, 1975 

Basts of PavTne.it ' , , ' " , ' '\ 
■, - irrespective pf the nature -of the committee's dtcJiSion resulting from review 
^ of the -subftiitted design, th6 Boar'd.of Edjjcatfon a^ees to pW the contractor 1 sum 
notttf exceed $560. 00 for tte preparation and'^uhmissioit bf tlft'ev^^on de^stgn 
proposal/ TRe contractor shall- submit tb 4he school district an invoice for • 




\ 



■vi6es perfoijpied and authorized reimbursable*^expenses.. The contractor is 
/ aUtho^;ized to.fcharge 22 percent of all invoiced costs in and for ihdirept costs. Each 
invoice shaU sJioW\fo\each einployee of the contractor that wprked on this project 



^uring^'the invoicfe period, the name and hours worked dii^tly ott'the project 
Also accompanying .the invoice-shall b^ evidence of reimbu^abie cpats paid - 
The contractor shall also provide a,detailed budge^'for the proposed • 
revaluation. The^names of those individuals who' will be primarily responsible - i 
for the evaluation slfaU be provided. The\chooi-distrl/t Wjll provide keypunching 
'services, copiputational equipment- tinie on an 366-50, aniiaccess to the ■ ' 
. Bio^Med ah^ SPSS Statical packages. 'Co^ta for these aptivi ties are not be * ' 

' , • ^ ■ . . . * ' ' <5 

. included. ^ . ' ' • . , ^ , > * ^ . ' ' 

, - ■ ■ ^ " !^ '■ .■ 

SchoonDistrfct of Metropol ^, , , . Evajuator's Anonymous,. Inc. " 
• B y • - _. By : " ' ■ 



Date • • A '. ^ ; Date' 



COMPREHENSIVE ' 
• ."bONTkACX FbR'AN 
f ■ - EVALUATION STUDY 



/ 

\ 



^ .This, agreement ent§red>into a^of this 1st day "of July, 1975 by and between tiie 

school district of -Metropol, hereinafter referred to as-the "di^strtct, " and ' ' 

^ . , • ^ - Evaljiator'^Anonymotis, .Inc. ' ' ' , 
,119 Bal Fontaine Road- 
^ » Juneau, Alaska" ' ■ ■ 

hereinafter referred ip as the "contracton " • ' 



. { 




V 
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Whereas, the design sjibmitted by the contractor for the evalua^on of the 
Preschool Concept D'eveldprnent Program (PCpP) has been modified by negotiation; 

- : Whereas, the Scho^Board of IVfetropoI approves the recommepiation of 
their negotiation committee to proceed with the evaluation; and, ' - ' 
Whereas, the, contractor is willing' to undertake this endeavor; * / 
Therefore , the parties do mutually agree as follows: ' / 

% \ 

1. Period of Contract . - \ , • 

I* 

^ This contract shall be in -force from 1 July, 1975 to 31 August, 1976 at which 
time the services agreed to are to have been provided an^ all products delivered 

as called for in this contract and in accord with the deadline included herein. 

- ^ ^ '' i • ' ' I , 

2. ^ .Scope of Services , , ' 

^ ' \ . " ' ' • ' ' ' ' ' ' 

' The contractor shall perfoni; those services necessary to accomplish thp 

* " (' * 

evaluation of the Preschool Concept Development Program -in accord with the 

/ • . . . ■ . 

,agreed upo"n design. Specifically, the contractor shall: 



1 L 

/ 



Select by use of procedures specified hereafter six kindergarten ciassrooAs 
to receive the PGDP instructional material and three kindergarten / 



classroonjs to serve as comparison classrooms." The classroomsi' are to- 
be jchosen so that they differ with respect to^ percent of the children 
who speak a native language in their home, as shown in Table 1. 
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TABLE 1 . . 
SAMPLING PLAN 



Percent 6f the students^* who.. • • . . ' ■ ' 

•speak aaati'vfelangifage V ■ 0~. 30% , . '3.1-65% '66 - 100%' 

P^DP Classrooms ' ■ 2 " . ' -2 - '2 '• 

\t Comparison Classrooms" r 4' '\ -V" 



4?» 



Qiirlng the school year each PCDP classroom is .to be observed for one ' ' 
< ' .hour per week on the average by two members of "the contractor's staff s • 
' ; . 'One halt of^is^; observation time willbe devoted to the 'use 6rthe*PCDP v^ 

Hhpl^mentation O^f^rydiiLOn Schedule.* .The rem^ini^g half witlbe uQa- t 
- strug^ned. Monthly prqgress reports will be^rivided to each PthV 

teacher oa^the ext^t toVhich he br sha is in con;tormity w^th* expected ' 



, PCDP teachingpractice. 



On 30 March, 1976 the contrdctor. ^iJl^submit ife^S^^i^rep^ '-' 

modificatioii made in,the PCd|^ Implementation Ob*seryation"Schedule,' /- ^ 

< \ ■ ' . •' ... -'4 ' • " 

as wen as a report on its (iharaQter^^ics. Included in th^jatter report' « > 

.• will be evidence 'Concerning its bbjectiVfty ^s-tvell as a cbrrelaf^nalr analysis 

■ •• • • '^f-v--^ ''Jf ■ 

of .the relationshiij bet^<^eep teacher self-repoi't data and the^ob'S&rvation data 

' ■■' ' ■■, ' ■ ' • . -' .' . 

• ,• . " , r - 

' A. final Report oh data collected by use of the scheduJfXvill b^ Included ip* . 
the evaluation report due on 31 August, ^976. • . , ' ^ 

.Dulling the months^of S#niber arid May, B^ch stude'n/in tji'e nine 
participating ciass^rooms wiH,be*l^*inister6^ 4hd SaltdpstairDoyelp^Jnient . ■ 



4f 



' Analysis Test by qualified test administrators employed by the 'cpntractor/. 
The test is to be individually administered. ' ' - ' '* 

The. contractor will be responsible for all statistical* analysis./ The May ' 

> .. . ' - . . ; * ' 

test date is to. be subjected to covariarice Analysis with the September scoijes' 

entered as the first CQvarf ate. The PCDP classrooms- will be contra3ted 

with the comparisori class fooms uising ihe classroom means of the May 

scores adjusted for* the pre-s'cores as ihh units of analysis/ Based on 

indices [of the'degrere of impletneiitatioa Qf the expected' teaching practice 

*/de^iVe<^.Jfrom th? observationaLl data,'- the PCDP .classroqtes are to be ' * 

• ^ » • ' / , • . ^ - ' 

, contrasted qn these same adjusted Qlassroom means. 

. ' ' /: • ■ • • I * ^ • ' ' ' * » • ^' 

• Th'e contractor will also analyze the -PCDP inaatery test data provided by 

the school district? for participating cMss^-ooms. Among the in&rm^tioa 

•' . ' ' ' « •' • * ' 

"to be presented \yill be the percent of'stiidents who have inastered the 

. • .'. • . 

various PCDP 'objectivps at the defined .criterion levels. These -data are "• 
alsp to be . presented so'as' t6 pennitcdinparison sitnong classrooms by 
'i-'implemiientatidn i^ndices.. V, " 



, ' .The contractor ^11 develop a tfeacher questionnairl fotro which wilLinclud^ 

. , a seetion perrfiittifflg teacher^ to rate themselves in terms of their " 

^ ^ * \ itnptementation of>GDP. '\The, que^tibn^^^^^^ ^' 

» , . * f ' . . , » * 

. ^rstrictfor review by Septemt^ 15,. 1975 jand revised afte^^ , 
V conoernirig changes. These' quest^rinaires aretto be admibistere^^ring 
. . November, February and May. Golfectiori gf this\qyest(on^^^ . - ' 

, / be the responsibility of the school (district., sthe contractor wiljpresent 'v- / 
frequency analyse^pf the questionnaire -datajn thfe ahabfVkfuafioh report. ' -""^ ' 



During the^cpu;rse of the eyaluaftioji, there will be tiw?6atypes.6f reports 
expected. Monthly progress, reports to the participating teachers on their ' 
implemeHtation olPCDP are to be providecUin Octi^ber, .November, Decen^b 
• January,. Februa^y;,*^ March and April. These repo'rts ar^to^^.kept / ' ' 
confidential, .i. e.-, only, the teacher in question.is to.fecgive a: copx; ' • 

. A technical' report ot.,the*PCpP Imprementatibn Observktlon ^chediU'e is^duV 
in March. Ten copie^ of this report are to be provided^^to the school • / 
district office. / . - * \ \ ' * *• 

The first draft of the fitial evaluation rejx)rt is du6 on ^||||Iuiy, 1976. ^en^ r 
copie^ are to be provided to. the school district office. ' After review by k . . * 
school distric.t appointed committee .discussed further In ^ectioa th|*ee of 
this contract, and subsequent Negotiations/ a final report will iD^ s^ibmitted 

oii 31 August, 1976. . Fifty copies are t<? Be provi'ded to the school district 

* ' * * * ' . - • • 

for distribution to .^e School Board dfficials, school district sbif,'^. 

parents and other interested partie§.;, J ' \\ . , * \ 



The final report is to preserve the confidentiality 'of all teachers and 

students^as stipulated iii section four of this contract. • * 

* * ' ♦ * 

3. Cooperation of thfe School District ' . . 

*» n ' ' * , 

. > * ^ ' ' . ' . ^ •* ' ^ 

> # \ * ' • * 

. ' ' . To facilitate the accomplishment of this evaluation, the school district 

commits it§elf to undertake and accomplish the following: . ' * . ' 

. *' . . ' ^ 

Sixe schooldistrict will prpvide the contractor with %.3chedule of the ' 
« * * * ' / ' ' 4 • 

^ planned classroofn use of tfiis't^CDP^ materials within ea6Ti participatfi^.. 

classfpom. This schedu-te will be updated on a weekly basis to injure! 
■ ' . . adequate obs^rVatidnal coverage. • , ' ' 

• . '■• ■ Mil. • -■ 
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' ■The school •district "will insure that theco^itr.actor has access to the . ' • V . '.' 
seFected classroom^ for obgervationaL purposes. To promote fidelity of • ' . 
ti»\iiif^rmation Obtained, the timing' of the contractor's ojiservational - \ . '* 
Visits tieed .not be pre-arr^ngq^J, with^Jhe te^chlr nor the school principal. " 

T^ -school district will cbllect and provide t^^^evcdntractor mastery test ^ ■ 
data for each student in th6 PCDP prograni. These" tests, of which &ere 
are three,' will pe administered .to the students -during November, February 
, and" May.. .^11 m'astet7 test d^ta to be analyzed will be provided to the • 

- ' ' • • * . • 

contractor by 15 June,"" 3.975, on keypunched- k column cards in a mutually - 
agreed upon format. . " . • ' ' ' a . • . " 



' The schooLdistrict will aollect teacher 'questionnaire data during the months 
' stip\ilated ii Secti6n"'2:' Theafe data: -Wiri be proyided to .lie contractor* by ■ . 
^5 Jm|||^1^76 on kSypufiched 80 column c^rd^ in a;mutually Agreed upon foimat* . 

^The schpel district will keypunch Jhe Saltonstall test d'afe, on 80 colUian - \ ^ " 
cards ^pcprding to a mutually agretecl. upo^ These card'g v^,ll*b'e ^. /• 

provided;fo the contractor.-^itHinUS calendar, days after the-.delivery of .' . . , 
the inforination'to be keypu^^ 

the sdjjooyi^trict will establish a rfeyiew coih^ _ ^- -• 

- t^r; • -15I?li ssa^ Frank, the Sugeriii|4nderit of the. S^Tetropol School District 
•.Dr. ,WllsbnBoyd, the;Director of Etementary. Instruction " • » ^ • 

- Ms^-JuUe Rath, the' Prjnfeip&l^ of the, Weathertown School . - • • • ■ 

%f-C' •• ■ ■ ■ ' • • 

•M!*' B^^hards,,:.a representative" chosen by the district's, kindergarten teachers 
pb^; Roland Qreen, Professor of Eduoational Psychology, Universtfy of Alaska ' 



THIS committee will review the 'technidal^isport, the teacher questionnaires 
and'the .first draft of the final report and retum,tlieir .comments- and ' \ t 
criticlsins to the contf actor, within 15 calendar days of receipt: The 
• .committee will also ser^i^e,as*'the negptiattng body fdr reaching [agreement . ' 
/ CQnfceming necessary changes lu thfee documents • , * , * 



A^(!lli<:irial Contract Pyovisions • • 
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A,' NoD-Dtscrimigatib n ^ ' . ' ' • ' • ' 

The contf Actt>r agrees that in- performing this contract, he or she shall -not ^ ' 

jo''" * * / ^ * * ' 

discriminate against a^y Wrker, employee or applicant; or anjr- member Qf * ' 
> Jhe.public, because of race, sex, creed,. <?ol6r or national origin,* nor otherwise 

. . ^ ' ' V ' . ^ L " ' " ' . ^ < , • ' 

commit. an unfai'flabpf' pra9ticef The^Oontractof -fijrther.agrees.that thig- ^ 

V clause will be incorporatetUi^^s^ entered into with suppliers of 

materials or services^ contrgtctors a^^*suEcoJibracto alMabor . - ' - 

organizations^rntshing sE^Ued, 'unskUl^dand -craft upjon skiil^cf khbr^ ' 

, or>ho m^ay perfoirm any such labor ^of seryiqes^h copmection with 'this - 



fconlxact/ 



B/"-,. Compliance With Laws ^' * ^ <. 

»* ' j -* ,^ , ^ 

The contractor shall dV.all tim^s observe and); cqmply .with all laws ^' ' • ^ 
• ' . - * ' ' . , - ' i-* ' ^ 

orflinaneps;and regulations of the'federal, -state, i6cal-#nd^<3ity gpvernnvent; " 
^jvfi^fi'iiiajjr in any 'manner affebt the performance of ttp^contradt:. / . ^ ' 



;C.( Insurance ; • 



The'eontractor will purchase and maintain -duritig "the iif& erf tills CDutr^ct - - ' 
msuranpe coverage which wtlJ>^tisfactorxly' insure. hijn.agaiti§t claims ' 
• and liabiIifies«whicTi could%^risfe b^caus.e^of the ejc^autlon of this :confract. • 



D. . ^COnflict of InteresL ''i'- * ^ ^ . ' > - 
* ' • T ^ ^ • J , 

' ,j ^ ' . < V* r • • » ^ ' / ' 

;. .No-member of the goyerning Body^of thejCity 'of iVfetlfo^^ 

^ , ^ ' % • r . " ' . / ^-''^ , 

' goyernm,ent'an<l.rio Qther.6fficer,[.employee^ or ageqt of tli6''City. or other* 
. ^ .unit of gov,erament wto^gicercis^s a'ay, functions or* respmisibilities in * 

• , conneetid.n with the cari)^vpg ont of ttie Ptojebt to; whicji.tiiis contract pertains, ' 
. ^\ shalLhave .^jr personal mter.est, di^§jct or indirect, iLthis .contract; The 
; contracijQr .covenants. that ,he piresentlx- has no interest, and' s'hali n'ot acquire ' ' 
any ipftefest," dii^etpr Kidibec ; * , • 

, -pertains yvhich- would;ponfli_ct in jaax faanHjsa:^.aegree witb^hf performance . ^ • 
- o£tos. services ihefeiinder. , The contractor Sirther coveaknts thit in the' • 
' . , performance of this confe*act'n<^^persontav/^ any such interest s^all be 
employed.,.- ' « ' . ^ » *' * * 

E. * ^ Confidentiality , ' ^ ^\ '* ' , * ^ 'I 
The c(J?itractor agrees to pres^We the confidentialily of all subjects 
.participating in thia evalyAtioa? nc teacher, student.or school will be \ * ' • * 
identified or identifiabrfe in the written oral reports provided to the- ^ 
district or iny other pariyV (The use o| fictitious names is permitted in \ / 
reports to illustrate individual u^e^ if that is deemed a desirable repc^ig V . . 
technique). ^ . • ,* ' * " . - • ^ , . 



F. 



Prtbr Review 



N.o written rfeport bf this evaluation wiilie' released by the contraptor tb 
any party, without the concurijence 6f the District ^R^view Committee/ The 
. district similarly agrees to's^^dure the approval of ^the i^'ontractpr prior to 
/'the dissemination of evaluation, findings. - > 
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-G. Other Research Interests ^ * ^ * * 

* '"^^ * * * - . ' \ ' ' . ^ ' ' 

The contractor is free to use the instruments, developed for use in this ' • 

project or the results derived therefrom in o^r pursuits as long as this 

use does not violate any other contractual provision, . ^ 

^ V ' V . • 

The contractor is similarly permitted to administer the '^Tea^jhtng Climate ' 

r ' \ • • 

Index" to those participating teachers who volunteer to respond to'cbVect 

c 

data for use in ongoing research in the area. The^aministration of this 

instrument is not to occur during normal school faiJurs; > . ' 

• * » 

.H. ^ PuBlication Rights ' ' . 

The .contractor is free to publish a report^of this evaluatioji'in a.professional' 
joumal.or to'present an account at a i^rof^ssional socifety meeting with the 
fqllo'^^ restrictions: . ' ' ' 

.1. The confidentiality of the' district, as we.il as that of. the ' ' 
^ • ; individuals "involved, is preserved. • ' / 

2. The article or presentation is first. submitted to the district ' 

« for prior approval (see section 4. F) ^ ' 

^ ; ^ • » . : ^ 

.1. ♦ Negotiation/Arbitration • / " 

■ ^ • / ' 

In the event of a disp'ute between the client and the contractor xioncerning 
>apy provision of this contract, that dispute will be submitted to a panel 
for arbitration. The pfnel will be ci^i^prised of: ' 
Dr. Harrison Greeley, Professor of Education, The University of Alaska, ^ 
$ Fairbanks / ' * 

Dr.,,James Evans, Senior, Research Associate, The Northwest Regional 



Dr. Michelle Williams, Associate Superintendent of Public Instruction, 

^ ' . ' ■ ' n 

Alaska Department of Educa]tion. ' ^ • "~ 

The findings of this panel wiH not be binding on either party. * 
J- ^ ! Termination of Contract ' , « » 

This agreenjent maj^ terminated by written mutual consent Iri/*^ 
the event of termination t^e contractor shall be reimbursed for its 

^ . . ■ " ^ / . ' : ' ' 

costs incurred to the date of termination. - 
JVIilestone Dates ' ' ' ' 

The dates of whict products must be deliyered and/or major activities must 
be concluded are shown below. ' • 

,0 Contract Agreement ' • 

0 Classrooms (PCDP and comparison) selected 
0 District provides client with classroom 
schedule:for use of pCDP—schedule to be 
updated by district on a Weekly hasis , • 
0 Draft of PCDP Implementation Questionnaire 

* * * • 

delivered to district for review ^ 

0 Observation of PCDP classroom hegins 

> • * <» 

0 Saltonstali Developmental Analysis Test. 

(SDAT) administered to PCDP and comparison 

classrooms " , • • 

0 Committee review of PCDP Implementation 

« 

: Questionnaire completed and delivered to 

contractor . * v ' 



lJuly, 1975 
. '30 'Xu|ust, 1975 
15 September, 1975 



30 September, 1975 



^ISOcJtober, 1975 



SlOctoblr, 1975 



15- November; '1975 



30 November, 1975 



[/5 December , ' 1975 



15 January, 1976 



.15 February; 1976 



' !29 Febrpary,' 1976 



15 March, 1976- 
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0 Mcmthly Implementation^Reportsi. provided to 
3Cm>- teachers " . ' 

A, ' * * u • * *" 

'o*Data from SDAT provided, to di&trlctfor " 
keypundhiug' ^: - . ' > ^ ./ 

*^ 0 Keypunched SPAT data delivered to cbntraotor 

^ ' ^ / ^ : \ ,^ * 

olPCDP Implementation Questionnaire. delivered 

* * "* 

to district^ . - 

oMojithly Implementation Rfej^orts'^t^vid^d to 
• ,PCDP teacjiers' 

, b PCDP" IinplemeYitation ^estionnaire adminis- 
. * tered to teachers by' district 
bPCDP Mastery Test I'data collected by district. 

< * 

oMon^ly Impleirientation Reports provided to 
.PCDP'teac^ers* ' ^ 
oI\^onthIy Impleinentati^^ Reports provided to 

PCDP teacher^ 
Q Monthly Impiemet^tati^^pports.provided* to 

•PCDP;teaclfers^ ' . - : 4^- " * - 

• . • • • ^ 

0 PCDP Implementation Questionnaire adnrinfs- I 

tered to teachers ^y district * ^* , « . * 

•* '•', ' * * • • ' 

bPCDP l^'astei?y\Test''2 data,-coll.ected 6y dtstricV ' . 

0 Monthly Implemehtation rieports provided^. . 'V 
K:dp. teachers- \/ 




o 



...i&May, 4976 



30 May, 1976 



15 June; 1976 • 
' . ^" IN 
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. 0 Draft of Interim Technical Rep6rji on the PCDP / ' 
Implenjtentalion Observation Schedule delivered 
to district ^ . . - - ^ • , ; 



0 Monthly Implementation Reports' proyid^d to \ *' 

PCDP.feacheps . /. ' • - 

0 Committee review of Jnterim R:eport delivered ^ •* 
to contractor ' ' . ' , ^ ♦ - 

/ . ■ - , ^ ; ■' : 

0 Monthly Implementation Repoirts provided to . " ; 

'• ■ ' • ■ • ' a" 
PCDP teachers " ' . . . , . . ' - J 

0 Final version, pf Intierim'^Report^delivered to \ ;-. ' 
. distiPipt-\.' \ . • /. %^ * 

o Saltoustall Developm'itital Analysis. Test ^(SD^^^kJ \ 
. administered to P.CDP and comparison , ^ 
classrooms " ^ ■ ^ - o x 

oPCDP Implemen^tioh Questi<inn4re*administered - 

> ' . , ' ■ . " < ^ , , . ' 

to teachers by district - * . ' ' , "i-: 
. ' • »* 

0 BCDP Mastery ,T^st 3 data calfected by district 
•oData frpm SDAT provided to* district for ;., t 

keypunching ' • , ' - , , , - ' 

biCeyptinc^ed'data proVi.ded by district lor all 
administra'tijxns of PCDP Implementation ' 



30 June, 3.9^6; 



i^'^ V*, ( <^e$tiontiai^^^^ and 'l^lastery ^sts 

)AT datal(}elivered to contractor . 



* Jo dr%,,;i976' \ ' ' " ' .Voraft off^kiSlj^rt due. / 



•ERIC' 



15 August, 1976 



31 August, Id'fG 



0 Committee review of Final Report delivered ' 
to' contractor. ' 



6.*^.' ^ Basi^ of Paytnent 



a Final RepDrt due 



' , The district agreed to pay' the sum of $14, 975, 00 to the contractor- for the , 
conduct of this evaluation. Payment is to be made .as follows:- • 




1975 



Octoher, 1975 



fl February,, -1976 

• 1 May, 1976 " 

AcQejgrtanqe ofFinai 
Repo?t^^^oi^4Bout 
30.Auguk,' 1976 . 



$2, 975^. 00 

$3,000.00^ ' 

v|3,~0b0.00 

$3,000;00 
J. 

$3,. poo. 00. ^ 



• The contractor .assures the district that the indirect costs charge^' against . 

this C9ntract wilLfiot exceed 20. 5% of the direct costs incurred. 

In thp event, of contract germination, the contractor will be r^m^urs^d 

' ' . *• ' • , ' .* ' 

for its costs, to the date oi termination. ... ' 



School pistricttof Metrbpol 

By ^ ^ . . 




' E valuer Anonymous, Inc. 



J 



Evaiuators Anonylous, Inc. ; 
119 Bal 'Fontaine/RoAd 
^ Juneau, ^aska-^ . , 



" Super;Lntendent L. K; Williams ' ' ' 

^Mdtrppol 'School District ^ ' ' ' • , / 

iletropol„ Alask^\ , * \ * 

Dear Superintendent .Williams :^ 

I am'^piaased^ to submit a contract document for evaluating the Pre-school 
concept Develdpment Program now in operation ,in the Metropol School 
Dist/ict.^^ Enc^losed are two copies of the contract for your signature. ' 
j^Please return onfe cdpj^ to me. * • ' - > 

We are looking forward to working with you .during the next severaL 
•months. If I may provide further information^, please^ do not' hesita^te 
to call upon me. ' ^ / 

Sincer^y, ' \^ » 



^Harriet Brand 
President 

HB/ph 



a'greem-ent 



• This agreeifhent is made and eatered into by and between the School 



District of Metropol, Alaska, hereinafter calle^the district, and Evahiator's 



Anonymous,, Inc. , hereinaftbr^alled the contractor. 
1.0 Purpose 



2:0 



The district M the coritraetoi^mutoally agre\ to carry out activities th^^^l' 
will result in*4e evaluation of thVPrerschool Concept Development 



C 

2.2 



' P.rogram (PCDP). 
The contractor will: ^ \ . * 

2. 1^ Devise a sampling strategy tq permit comp'apson of .classrooms in 
which PCDP\^ used and other similar classrooms. 
Devise an observational schedule for u§e in classrqgm? using 
.PCPP in order to determine the extent to which the program is 
implemented as. intended. 

Observe each PCDP classroom on a one hour per w4k basis and 
provide- a monthly report to each participating teacher. 
Administer the Saltonstall Developmental Analysis Test to 
classrooms using PCDP and comparison classrbomE in ' / ' 
September, 1975 and May, 1976. . 

Develop a questionnaire to elicit information about program use 

for teachers using PCDP. , ^ , " 

- — ' \ 

Provide a technical report' on the observation schedule in 
Mar.ch, 1976. . a 



2.3 



2.4' 



2.5 
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2. 7 An^yze all data, :ii;iclu'dipg that provided by the district. • 

2.8 Prepare a draft copy of the.-final report by '30 July, 1976. ; "' 

2. 9 Prepare a revised final report by 30 August, 1976. 
The district will: , ^ - ,^ 

1 Provide the contif actors with a*schedule of classroom use of PtJDP,, ; 
. 3. 2 . Collect guesti*oniia\j;e data from^achers using PCDP during - ^ ' 
November, 1975; February, 1976 and May, 1976, . " • . * . 
^. 3 Collect Student data on PCDP mastery tests, according to the' 
' following schedule: - - ^ . ^ * , 

Mastery Test 1 ' , November, 1975 ^ '\ ^ 

Miastery Test 2. , February, 1976 ; ' 

' ^ ' Maste'ry Test 3 May, 1976 ^' - . ^ 



3. 4 keypunch all data on 80 column cards 'in a mutually agreed upon'' 
format. v ' ; . * / 



\ 



3. 5t ' Provide free access to the district's computer system fpt data 

analysis^ ' . > 

3. 6 Return all documents submitteji for review within 15 days of 

' receipt* • ^ ' ^ / ^ 0 • * 

3. 7 Provide, a sum of $14l 975. 00 to the contractbr for tiie conduct of 



this evaluation. 



Conditions 

4. 1 The peridd of performance shall be irord July^ 1975 to 30. August, 
' 1976. * ^ ' . / • 



• " • ^-"f . . '^^^ 1?® * fixed.pric^fe contract in the, amount of $14, 975. 00 . 
: -"^ for the pecforiijance • df wcu-k^stipulated' In paragraphs 2. % thr ou^' 



4. 3^ / krty'cdpies 'of felijiaVx^pbff shall be-deUvered to the district 
without restrictidn as to Wu^e-.- ,4 addition^ the cohtractoV shaU 
. have Unrestrjcted' use-of the contents of ike producisr either in fee' 

- ejctant-or an adapfed foi^, for its continued use -in the field of 
■ . education, with or ;without reference to fee district L dete^ined- 
' ' * by theiiistrict . • ' ^ * ' 

^.4'-.. The. payment schedule ^0-tHe contractor wiU.be as follows: 



1-July , 1975 • , . • . > '•: $2,-975: 00 
.I'.pctober; 19.75 . , ^^S, OOO. 00 



1 Febnfdry, 1976 
IMay, 1976 



$3,ooo."o,b 

J% 6,00. 00 



. Acceptanoe.pf Tinal 
Report on .or about ^ - 
50 August, 1976' . . 



\ ■ 

■\ ' $3y000. 00 

\ . . • 



4.5 



1 



TMi agreement may He tefminated by ihlten mutual ooiisent. In, 
the event of termination fee contractotj sh^il be reimbuVsed for 
its costs incurred to the date of termilation. • '• 



School District of Metropol 

By ' 



0 

Date 



Evaluator's Acjonymous, Inc.'' 
'By_ 
Date 



w ■ 
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